Machine Learning Engineer Nanodegree

Capstone

Project- Corn Commodity Futures Price Predictor

In this project we will try to predict closing weekly price of Corn Commodity Futures. In order to perform this prediction we will create a dataset that includes weekly Corn Futures closing prices as well as Long Open Interest and Short Open Interest of Processors/Users( sometimes they are called Commercials) from COT reports and by using this dataset we will try to predict next week’s prices.

1. Data Sets

Historical Futures Prices: Corn Futures, Continuous Contract #1. Non-adjusted price based on spot-month continuous contract calculations. Raw data from CME: Can be found here
Commitment of Traders - CORN (CBT) - Futures Only (002602) Can be found here

Data has been downloaded and stored in \Data folder:

  • .\data\CHRIS-CME_C1.csv - Corn Futures Prices data
  • .\data\CFTC-002602_F_ALL.csv - Commitment of Traders data
In [1]:
import warnings
warnings.filterwarnings('ignore')
In [2]:
import pandas as pd
import numpy as np
from IPython.core.display import display, HTML
pd.options.display.max_colwidth = 500  # You need this, otherwise pandas
# will limit your HTML strings to 50 characters
pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', None)
pd.options.mode.chained_assignment = None  # default='warn'
from matplotlib import pyplot
from sklearn.preprocessing import MinMaxScaler
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import LSTM
from math import sqrt
from numpy import concatenate
from sklearn.metrics import mean_squared_error
import matplotlib.pyplot as plt
from plotly.offline import download_plotlyjs, init_notebook_mode, plot, iplot
import cufflinks as cf
import plotly.tools as tls
init_notebook_mode(connected=True)
cf.go_offline()
Using TensorFlow backend.
C:\Users\zilvi\Anaconda3\envs\zil_tensorflow\lib\site-packages\plotly\graph_objs\_deprecations.py:558: DeprecationWarning:

plotly.graph_objs.YAxis is deprecated.
Please replace it with one of the following more specific types
  - plotly.graph_objs.layout.YAxis
  - plotly.graph_objs.layout.scene.YAxis


C:\Users\zilvi\Anaconda3\envs\zil_tensorflow\lib\site-packages\plotly\graph_objs\_deprecations.py:531: DeprecationWarning:

plotly.graph_objs.XAxis is deprecated.
Please replace it with one of the following more specific types
  - plotly.graph_objs.layout.XAxis
  - plotly.graph_objs.layout.scene.XAxis


2. Prepare and Explore Data

In [3]:
df_fut_orig = pd.read_csv('data\CHRIS-CME_C1.csv')
df_fut_orig.head(n=5)
Out[3]:
Date Open High Low Last Change Settle Volume Previous_Day_Open_Interest
0 2018-07-10 344.25 344.75 336.25 339.50 6.00 339.75 2668.0 2186.0
1 2018-07-09 346.00 348.50 342.50 346.00 6.00 345.75 3190.0 2969.0
2 2018-07-06 342.00 352.25 342.00 350.75 8.25 351.75 3068.0 3959.0
3 2018-07-05 345.50 348.75 341.50 342.50 0.75 343.50 3302.0 4812.0
4 2018-07-03 340.25 345.25 339.25 343.25 5.25 342.75 3048.0 5687.0
In [4]:
# Display a description of the dataset
display(df_fut_orig.describe())
Open High Low Last Change Settle Volume Previous_Day_Open_Interest
count 3033.000000 3034.000000 3034.000000 3034.000000 1081.000000 3034.000000 3034.000000 3034.00000
mean 457.095038 462.322924 451.795485 456.920040 3.950324 456.979318 103905.200396 352140.90145
std 140.338892 142.056030 138.436196 140.243019 3.415126 140.204571 73993.219920 248565.85531
min 219.000000 220.750000 216.750000 219.000000 0.000000 219.000000 0.000000 107.00000
25% 360.000000 363.000000 356.250000 359.500000 1.500000 359.750000 40172.750000 107559.25000
50% 388.500000 392.000000 383.500000 388.750000 3.000000 389.000000 102567.000000 365073.00000
75% 565.500000 573.562500 557.375000 564.625000 5.500000 564.625000 152391.250000 556408.50000
max 830.250000 843.750000 822.750000 831.250000 30.750000 831.250000 538170.000000 858696.00000
In [5]:
df_fut_orig['Date'] = pd.to_datetime(df_fut_orig['Date'])
df_fut_orig.set_index('Date',inplace=True)
df_fut_orig = df_fut_orig.sort_values('Date')

Plot Corn Futures Price Series using Plotly

In [6]:
%load_ext autoreload
%autoreload 2
import visuals

visuals.plot_original_price_series(df_fut_orig)

Seems there are some rows where Volume=0, lets find out more about these rows

In [7]:
df_fut_orig[df_fut_orig['Volume']<1]
Out[7]:
Open High Low Last Change Settle Volume Previous_Day_Open_Interest
Date
2007-04-05 359.75 367.50 357.25 366.00 NaN 366.00 0.0 354349.0
2012-04-06 658.25 658.25 658.25 658.25 NaN 658.25 0.0 401521.0
2015-04-03 386.50 386.50 386.50 386.50 NaN 386.50 0.0 470964.0

Since we will resample daily prices into weekly prices , lets drop those rows.

In [8]:
# drop outliers
df_fut_orig.drop(df_fut_orig[df_fut_orig.Volume<1].index, inplace=True)
In [9]:
df_cot_orig = pd.read_csv('data\CFTC-002602_F_ALL.csv')
display(df_cot_orig.head())
Date Open_Interest Producer_Merchant_Processor_User_Longs Producer_Merchant_Processor_User_Shorts Swap Dealer Longs Swap Dealer Shorts Swap Dealer Spreads Money Manager Longs Money Manager Shorts Money Manager Spreads Other Reportable Longs Other Reportable Shorts Other Reportable Spreads Total Reportable Longs Total Reportable Shorts Non Reportable Longs Non Reportable Shorts
0 2018-07-10 1818055.0 500172.0 750062.0 208128.0 39513.0 99477.0 263353.0 404297.0 154286.0 320946.0 70682.0 98709.0 1645071.0 1617026.0 172984.0 201029.0
1 2018-07-03 1830330.0 484257.0 773851.0 210341.0 36927.0 100340.0 274795.0 382191.0 149756.0 322256.0 66508.0 119627.0 1661372.0 1629200.0 168958.0 201130.0
2 2018-06-26 1885804.0 513100.0 840177.0 223131.0 32763.0 91972.0 287061.0 377825.0 153461.0 330396.0 58283.0 116745.0 1715866.0 1671226.0 169938.0 214578.0
3 2018-06-19 1992169.0 525197.0 920764.0 222105.0 41144.0 99285.0 299377.0 356828.0 163454.0 379025.0 56652.0 135078.0 1823521.0 1773205.0 168648.0 218964.0
4 2018-06-12 1963233.0 488666.0 917204.0 235249.0 37674.0 93281.0 292054.0 304292.0 172623.0 363918.0 65030.0 147098.0 1792889.0 1737202.0 170344.0 226031.0
In [10]:
display(df_cot_orig.describe())
Open_Interest Producer_Merchant_Processor_User_Longs Producer_Merchant_Processor_User_Shorts Swap Dealer Longs Swap Dealer Shorts Swap Dealer Spreads Money Manager Longs Money Manager Shorts Money Manager Spreads Other Reportable Longs Other Reportable Shorts Other Reportable Spreads Total Reportable Longs Total Reportable Shorts Non Reportable Longs Non Reportable Shorts
count 6.310000e+02 631.000000 6.310000e+02 631.000000 631.000000 631.000000 631.000000 631.000000 631.000000 631.000000 631.000000 631.000000 6.310000e+02 6.310000e+02 631.000000 631.000000
mean 1.292201e+06 270795.049128 6.268425e+05 290792.497623 20337.034865 33260.068146 236884.269414 137472.426307 94546.356577 140931.890650 70914.334390 85505.109350 1.152715e+06 1.068878e+06 139485.541997 223322.976228
std 2.095471e+05 68976.221600 1.554272e+05 53203.484072 18944.008732 22912.567257 67454.195123 109465.025186 32739.133163 51939.690903 26360.863384 29682.425476 1.939790e+05 2.060080e+05 23718.957966 29824.710288
min 7.482520e+05 102373.000000 2.972960e+05 186981.000000 0.000000 4397.000000 96989.000000 6714.000000 29130.000000 49809.000000 25905.000000 27592.000000 6.379810e+05 5.689510e+05 78578.000000 156086.000000
25% 1.192226e+06 226595.000000 5.235930e+05 255196.500000 6524.000000 13978.000000 186366.500000 47947.000000 72018.500000 104764.000000 53331.000000 62690.000000 1.055362e+06 9.573815e+05 121829.500000 198860.500000
50% 1.301506e+06 262823.000000 6.112810e+05 276337.000000 15239.000000 27209.000000 225682.000000 95548.000000 91850.000000 140343.000000 66261.000000 82705.000000 1.166372e+06 1.067548e+06 136966.000000 227337.000000
75% 1.398275e+06 314224.000000 7.058555e+05 321265.500000 28178.000000 48009.500000 287331.000000 211154.000000 113803.000000 175846.000000 83448.500000 106077.500000 1.247976e+06 1.180280e+06 153542.500000 246903.000000
max 1.992169e+06 525197.000000 1.001517e+06 422803.000000 95591.000000 113775.000000 431569.000000 447470.000000 231064.000000 379025.000000 173322.000000 181385.000000 1.825238e+06 1.773205e+06 206821.000000 293948.000000

Drop unnecessary columns columns and resample data

In [11]:
df_fut=df_fut_orig.drop(columns=[clmn for i,clmn in enumerate(df_fut_orig.columns) if i not in [5,6,7] ],axis=1)
display(df_fut.head())
Settle Volume Previous_Day_Open_Interest
Date
2006-06-16 235.50 56486.0 203491.0
2006-06-19 229.75 51299.0 190044.0
2006-06-20 229.75 41605.0 175859.0
2006-06-21 232.75 29803.0 162348.0
2006-06-22 230.50 28687.0 147658.0
In [12]:
s_settle =df_fut['Settle'].resample('W').last()
s_volume =df_fut['Volume'].resample('W').last()
df_fut_weekly = pd.concat([s_settle,s_volume], axis=1)
display(df_fut_weekly.head())
Settle Volume
Date
2006-06-18 235.50 56486.0
2006-06-25 228.25 28361.0
2006-07-02 235.50 30519.0
2006-07-09 241.00 13057.0
2006-07-16 253.50 2460.0
In [13]:
df_cot=df_cot_orig.drop(columns=[clmn for i,clmn in enumerate(df_cot_orig.columns) if i not in [0,1,2,3 ]],axis=1)
df_cot.rename(index=str, columns={"Producer_Merchant_Processor_User_Longs": "Longs", \
                                  "Producer_Merchant_Processor_User_Shorts": "Shorts"},inplace=True)
df_cot['Date'] = pd.to_datetime(df_cot['Date'])
df_cot.set_index('Date',inplace=True)
display(df_cot.head())
Open_Interest Longs Shorts
Date
2018-07-10 1818055.0 500172.0 750062.0
2018-07-03 1830330.0 484257.0 773851.0
2018-06-26 1885804.0 513100.0 840177.0
2018-06-19 1992169.0 525197.0 920764.0
2018-06-12 1963233.0 488666.0 917204.0
In [14]:
s_longs =df_cot['Longs'].resample('W').last()
s_shorts =df_cot['Shorts'].resample('W').last()
s_open_interest =df_cot['Open_Interest'].resample('W').last()
df_cot_weekly = pd.concat([s_open_interest,s_longs, s_shorts], axis=1)
display(df_cot_weekly.head(5))
Open_Interest Longs Shorts
Date
2006-06-18 1320155.0 209662.0 699163.0
2006-06-25 1321520.0 224476.0 666688.0
2006-07-02 1329400.0 234769.0 645735.0
2006-07-09 1327482.0 220552.0 648405.0
2006-07-16 1333225.0 216968.0 673110.0
In [15]:
df_weekly = pd.merge(df_fut_weekly,df_cot_weekly, on='Date')
display(df_weekly.head(5))
Settle Volume Open_Interest Longs Shorts
Date
2006-06-18 235.50 56486.0 1320155.0 209662.0 699163.0
2006-06-25 228.25 28361.0 1321520.0 224476.0 666688.0
2006-07-02 235.50 30519.0 1329400.0 234769.0 645735.0
2006-07-09 241.00 13057.0 1327482.0 220552.0 648405.0
2006-07-16 253.50 2460.0 1333225.0 216968.0 673110.0
In [16]:
# Display a description of the dataset
display(df_weekly.describe())
Settle Volume Open_Interest Longs Shorts
count 631.000000 631.000000 6.310000e+02 631.000000 6.310000e+02
mean 456.978605 100835.204437 1.292201e+06 270795.049128 6.268425e+05
std 140.242112 72466.341538 2.095471e+05 68976.221600 1.554272e+05
min 219.750000 132.000000 7.482520e+05 102373.000000 2.972960e+05
25% 359.500000 34822.500000 1.192226e+06 226595.000000 5.235930e+05
50% 389.250000 101209.000000 1.301506e+06 262823.000000 6.112810e+05
75% 560.375000 150341.000000 1.398275e+06 314224.000000 7.058555e+05
max 824.500000 369522.000000 1.992169e+06 525197.000000 1.001517e+06
In [17]:
# rest index since we need row numbers for splitting
df_weekly_idx_date=df_weekly.copy()
df_weekly.reset_index(inplace=True)

3. Visualise Data

In [18]:
%load_ext autoreload
%autoreload 2
import visuals

visuals.plot_weekly_combined_series_by_date(df_weekly)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
In [19]:
%load_ext autoreload
%autoreload 2
import visuals

visuals.plot_weekly_combined_series_by_trading_week(df_weekly)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
In [20]:
%load_ext autoreload
%autoreload 2
import visuals
visuals.plot_grouped_by_year_data(df_weekly_idx_date,"Stacked Plots of Price by Year")
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
In [21]:
%load_ext autoreload
%autoreload 2
import visuals
visuals.lag_plot(df_weekly,"Lag Plot")
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload

4. Normalise the data using minmaxscaler

In [22]:
scaler = MinMaxScaler(feature_range=(0, 1))
values = df_weekly.loc[:, df_weekly.columns != 'Date'].values
scaled = scaler.fit_transform(values)

5. Split data into training, validation and test sets

In [23]:
validation_start=df_weekly[df_weekly['Date'] >= pd.to_datetime('2017-01-01')].index[0]
testing_start=df_weekly[df_weekly['Date'] >= pd.to_datetime('2018-01-01')].index[0]
In [24]:
print("validation start",validation_start)
print("testing start",testing_start)
validation start 550
testing start 603
In [25]:
# print data to double check
#print(df_weekly.iloc[validation_start])
#print(df_weekly.iloc[testing_start])
In [26]:
%load_ext autoreload
%autoreload 2
import data_preparer
reframed = data_preparer.series_to_supervised(scaled, 1, 1)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
In [27]:
# drop columns we don't want to predict
reframed.drop(reframed.columns[[6,7,8,9]], axis=1, inplace=True)
In [28]:
display(reframed.head())
var1(t-1) var2(t-1) var3(t-1) var4(t-1) var5(t-1) var1(t)
1 0.026044 0.152560 0.459760 0.253744 0.570655 0.014055
2 0.014055 0.076421 0.460857 0.288780 0.524540 0.026044
3 0.026044 0.082263 0.467192 0.313123 0.494786 0.035138
4 0.035138 0.034990 0.465650 0.279499 0.498578 0.055808
5 0.055808 0.006302 0.470267 0.271023 0.533659 0.028938

6. Define and Fit Model

In [66]:
%load_ext autoreload
%autoreload 2
import data_preparer
train_X, train_y, validation_X, validation_y,test_X, test_y = data_preparer.split_data(reframed,validation_start,testing_start)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
In [67]:
%load_ext autoreload
%autoreload 2
import models
model,history=models.basic_lstm_model(train_X,train_y,validation_X,validation_y)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
Train on 550 samples, validate on 53 samples
Epoch 1/500
 - 14s - loss: 0.5592 - val_loss: 0.3919
Epoch 2/500
 - 0s - loss: 0.5173 - val_loss: 0.3437
Epoch 3/500
 - 0s - loss: 0.4764 - val_loss: 0.2974
Epoch 4/500
 - 0s - loss: 0.4374 - val_loss: 0.2533
Epoch 5/500
 - 0s - loss: 0.4009 - val_loss: 0.2120
Epoch 6/500
 - 0s - loss: 0.3678 - val_loss: 0.1729
Epoch 7/500
 - 0s - loss: 0.3363 - val_loss: 0.1351
Epoch 8/500
 - 0s - loss: 0.3058 - val_loss: 0.0985
Epoch 9/500
 - 0s - loss: 0.2770 - val_loss: 0.0638
Epoch 10/500
 - 0s - loss: 0.2515 - val_loss: 0.0362
Epoch 11/500
 - 0s - loss: 0.2331 - val_loss: 0.0267
Epoch 12/500
 - 0s - loss: 0.2226 - val_loss: 0.0308
Epoch 13/500
 - 0s - loss: 0.2165 - val_loss: 0.0382
Epoch 14/500
 - 0s - loss: 0.2126 - val_loss: 0.0453
Epoch 15/500
 - 0s - loss: 0.2100 - val_loss: 0.0516
Epoch 16/500
 - 0s - loss: 0.2083 - val_loss: 0.0560
Epoch 17/500
 - 0s - loss: 0.2071 - val_loss: 0.0595
Epoch 18/500
 - 0s - loss: 0.2062 - val_loss: 0.0622
Epoch 19/500
 - 0s - loss: 0.2054 - val_loss: 0.0641
Epoch 20/500
 - 0s - loss: 0.2047 - val_loss: 0.0655
Epoch 21/500
 - 0s - loss: 0.2041 - val_loss: 0.0665
Epoch 22/500
 - 0s - loss: 0.2036 - val_loss: 0.0669
Epoch 23/500
 - 0s - loss: 0.2031 - val_loss: 0.0670
Epoch 24/500
 - 0s - loss: 0.2026 - val_loss: 0.0672
Epoch 25/500
 - 0s - loss: 0.2021 - val_loss: 0.0674
Epoch 26/500
 - 0s - loss: 0.2016 - val_loss: 0.0675
Epoch 27/500
 - 0s - loss: 0.2012 - val_loss: 0.0676
Epoch 28/500
 - 0s - loss: 0.2007 - val_loss: 0.0677
Epoch 29/500
 - 0s - loss: 0.2002 - val_loss: 0.0678
Epoch 30/500
 - 0s - loss: 0.1998 - val_loss: 0.0677
Epoch 31/500
 - 0s - loss: 0.1994 - val_loss: 0.0675
Epoch 32/500
 - 0s - loss: 0.1989 - val_loss: 0.0672
Epoch 33/500
 - 0s - loss: 0.1985 - val_loss: 0.0669
Epoch 34/500
 - 0s - loss: 0.1980 - val_loss: 0.0665
Epoch 35/500
 - 0s - loss: 0.1976 - val_loss: 0.0662
Epoch 36/500
 - 0s - loss: 0.1972 - val_loss: 0.0659
Epoch 37/500
 - 0s - loss: 0.1967 - val_loss: 0.0655
Epoch 38/500
 - 0s - loss: 0.1963 - val_loss: 0.0652
Epoch 39/500
 - 0s - loss: 0.1959 - val_loss: 0.0648
Epoch 40/500
 - 0s - loss: 0.1954 - val_loss: 0.0644
Epoch 41/500
 - 0s - loss: 0.1950 - val_loss: 0.0639
Epoch 42/500
 - 0s - loss: 0.1945 - val_loss: 0.0634
Epoch 43/500
 - 0s - loss: 0.1941 - val_loss: 0.0630
Epoch 44/500
 - 0s - loss: 0.1936 - val_loss: 0.0626
Epoch 45/500
 - 0s - loss: 0.1931 - val_loss: 0.0621
Epoch 46/500
 - 0s - loss: 0.1927 - val_loss: 0.0616
Epoch 47/500
 - 0s - loss: 0.1922 - val_loss: 0.0612
Epoch 48/500
 - 0s - loss: 0.1917 - val_loss: 0.0608
Epoch 49/500
 - 0s - loss: 0.1912 - val_loss: 0.0604
Epoch 50/500
 - 0s - loss: 0.1907 - val_loss: 0.0601
Epoch 51/500
 - 0s - loss: 0.1902 - val_loss: 0.0598
Epoch 52/500
 - 0s - loss: 0.1896 - val_loss: 0.0595
Epoch 53/500
 - 0s - loss: 0.1891 - val_loss: 0.0592
Epoch 54/500
 - 0s - loss: 0.1885 - val_loss: 0.0589
Epoch 55/500
 - 0s - loss: 0.1879 - val_loss: 0.0587
Epoch 56/500
 - 0s - loss: 0.1873 - val_loss: 0.0585
Epoch 57/500
 - 0s - loss: 0.1867 - val_loss: 0.0583
Epoch 58/500
 - 0s - loss: 0.1860 - val_loss: 0.0582
Epoch 59/500
 - 0s - loss: 0.1854 - val_loss: 0.0582
Epoch 60/500
 - 0s - loss: 0.1847 - val_loss: 0.0582
Epoch 61/500
 - 0s - loss: 0.1840 - val_loss: 0.0582
Epoch 62/500
 - 0s - loss: 0.1832 - val_loss: 0.0580
Epoch 63/500
 - 0s - loss: 0.1825 - val_loss: 0.0578
Epoch 64/500
 - 0s - loss: 0.1817 - val_loss: 0.0577
Epoch 65/500
 - 0s - loss: 0.1808 - val_loss: 0.0577
Epoch 66/500
 - 0s - loss: 0.1800 - val_loss: 0.0576
Epoch 67/500
 - 0s - loss: 0.1791 - val_loss: 0.0574
Epoch 68/500
 - 0s - loss: 0.1781 - val_loss: 0.0572
Epoch 69/500
 - 0s - loss: 0.1772 - val_loss: 0.0570
Epoch 70/500
 - 0s - loss: 0.1762 - val_loss: 0.0565
Epoch 71/500
 - 0s - loss: 0.1752 - val_loss: 0.0560
Epoch 72/500
 - 0s - loss: 0.1741 - val_loss: 0.0552
Epoch 73/500
 - 0s - loss: 0.1730 - val_loss: 0.0546
Epoch 74/500
 - 0s - loss: 0.1719 - val_loss: 0.0541
Epoch 75/500
 - 0s - loss: 0.1707 - val_loss: 0.0534
Epoch 76/500
 - 0s - loss: 0.1695 - val_loss: 0.0527
Epoch 77/500
 - 0s - loss: 0.1683 - val_loss: 0.0522
Epoch 78/500
 - 0s - loss: 0.1670 - val_loss: 0.0518
Epoch 79/500
 - 0s - loss: 0.1656 - val_loss: 0.0515
Epoch 80/500
 - 0s - loss: 0.1642 - val_loss: 0.0510
Epoch 81/500
 - 0s - loss: 0.1628 - val_loss: 0.0505
Epoch 82/500
 - 0s - loss: 0.1613 - val_loss: 0.0499
Epoch 83/500
 - 0s - loss: 0.1597 - val_loss: 0.0495
Epoch 84/500
 - 0s - loss: 0.1581 - val_loss: 0.0491
Epoch 85/500
 - 0s - loss: 0.1564 - val_loss: 0.0489
Epoch 86/500
 - 0s - loss: 0.1546 - val_loss: 0.0483
Epoch 87/500
 - 0s - loss: 0.1528 - val_loss: 0.0477
Epoch 88/500
 - 0s - loss: 0.1509 - val_loss: 0.0471
Epoch 89/500
 - 0s - loss: 0.1489 - val_loss: 0.0469
Epoch 90/500
 - 0s - loss: 0.1469 - val_loss: 0.0470
Epoch 91/500
 - 0s - loss: 0.1447 - val_loss: 0.0471
Epoch 92/500
 - 0s - loss: 0.1425 - val_loss: 0.0472
Epoch 93/500
 - 0s - loss: 0.1402 - val_loss: 0.0470
Epoch 94/500
 - 0s - loss: 0.1378 - val_loss: 0.0468
Epoch 95/500
 - 0s - loss: 0.1353 - val_loss: 0.0466
Epoch 96/500
 - 0s - loss: 0.1328 - val_loss: 0.0460
Epoch 97/500
 - 0s - loss: 0.1303 - val_loss: 0.0456
Epoch 98/500
 - 0s - loss: 0.1277 - val_loss: 0.0452
Epoch 99/500
 - 0s - loss: 0.1250 - val_loss: 0.0446
Epoch 100/500
 - 0s - loss: 0.1222 - val_loss: 0.0442
Epoch 101/500
 - 0s - loss: 0.1193 - val_loss: 0.0440
Epoch 102/500
 - 0s - loss: 0.1164 - val_loss: 0.0437
Epoch 103/500
 - 0s - loss: 0.1134 - val_loss: 0.0435
Epoch 104/500
 - 0s - loss: 0.1103 - val_loss: 0.0435
Epoch 105/500
 - 0s - loss: 0.1071 - val_loss: 0.0438
Epoch 106/500
 - 0s - loss: 0.1039 - val_loss: 0.0443
Epoch 107/500
 - 0s - loss: 0.1007 - val_loss: 0.0448
Epoch 108/500
 - 0s - loss: 0.0976 - val_loss: 0.0453
Epoch 109/500
 - 0s - loss: 0.0945 - val_loss: 0.0459
Epoch 110/500
 - 0s - loss: 0.0915 - val_loss: 0.0467
Epoch 111/500
 - 0s - loss: 0.0886 - val_loss: 0.0477
Epoch 112/500
 - 0s - loss: 0.0856 - val_loss: 0.0486
Epoch 113/500
 - 0s - loss: 0.0827 - val_loss: 0.0500
Epoch 114/500
 - 0s - loss: 0.0804 - val_loss: 0.0517
Epoch 115/500
 - 0s - loss: 0.0783 - val_loss: 0.0534
Epoch 116/500
 - 0s - loss: 0.0761 - val_loss: 0.0539
Epoch 117/500
 - 0s - loss: 0.0747 - val_loss: 0.0551
Epoch 118/500
 - 0s - loss: 0.0732 - val_loss: 0.0562
Epoch 119/500
 - 0s - loss: 0.0717 - val_loss: 0.0570
Epoch 120/500
 - 0s - loss: 0.0705 - val_loss: 0.0583
Epoch 121/500
 - 0s - loss: 0.0693 - val_loss: 0.0590
Epoch 122/500
 - 0s - loss: 0.0679 - val_loss: 0.0589
Epoch 123/500
 - 0s - loss: 0.0670 - val_loss: 0.0597
Epoch 124/500
 - 0s - loss: 0.0660 - val_loss: 0.0602
Epoch 125/500
 - 0s - loss: 0.0649 - val_loss: 0.0599
Epoch 126/500
 - 0s - loss: 0.0640 - val_loss: 0.0601
Epoch 127/500
 - 0s - loss: 0.0630 - val_loss: 0.0599
Epoch 128/500
 - 0s - loss: 0.0621 - val_loss: 0.0597
Epoch 129/500
 - 0s - loss: 0.0614 - val_loss: 0.0594
Epoch 130/500
 - 0s - loss: 0.0606 - val_loss: 0.0593
Epoch 131/500
 - 0s - loss: 0.0598 - val_loss: 0.0592
Epoch 132/500
 - 0s - loss: 0.0590 - val_loss: 0.0587
Epoch 133/500
 - 0s - loss: 0.0583 - val_loss: 0.0588
Epoch 134/500
 - 0s - loss: 0.0576 - val_loss: 0.0586
Epoch 135/500
 - 0s - loss: 0.0568 - val_loss: 0.0582
Epoch 136/500
 - 0s - loss: 0.0561 - val_loss: 0.0579
Epoch 137/500
 - 0s - loss: 0.0554 - val_loss: 0.0573
Epoch 138/500
 - 0s - loss: 0.0548 - val_loss: 0.0567
Epoch 139/500
 - 0s - loss: 0.0542 - val_loss: 0.0563
Epoch 140/500
 - 0s - loss: 0.0536 - val_loss: 0.0562
Epoch 141/500
 - 0s - loss: 0.0529 - val_loss: 0.0552
Epoch 142/500
 - 0s - loss: 0.0524 - val_loss: 0.0547
Epoch 143/500
 - 0s - loss: 0.0518 - val_loss: 0.0544
Epoch 144/500
 - 0s - loss: 0.0512 - val_loss: 0.0539
Epoch 145/500
 - 0s - loss: 0.0507 - val_loss: 0.0535
Epoch 146/500
 - 0s - loss: 0.0501 - val_loss: 0.0527
Epoch 147/500
 - 0s - loss: 0.0496 - val_loss: 0.0520
Epoch 148/500
 - 0s - loss: 0.0491 - val_loss: 0.0516
Epoch 149/500
 - 0s - loss: 0.0485 - val_loss: 0.0511
Epoch 150/500
 - 0s - loss: 0.0480 - val_loss: 0.0505
Epoch 151/500
 - 0s - loss: 0.0476 - val_loss: 0.0502
Epoch 152/500
 - 0s - loss: 0.0471 - val_loss: 0.0496
Epoch 153/500
 - 0s - loss: 0.0466 - val_loss: 0.0488
Epoch 154/500
 - 0s - loss: 0.0461 - val_loss: 0.0483
Epoch 155/500
 - 0s - loss: 0.0457 - val_loss: 0.0474
Epoch 156/500
 - 0s - loss: 0.0453 - val_loss: 0.0467
Epoch 157/500
 - 0s - loss: 0.0449 - val_loss: 0.0461
Epoch 158/500
 - 0s - loss: 0.0445 - val_loss: 0.0451
Epoch 159/500
 - 0s - loss: 0.0441 - val_loss: 0.0445
Epoch 160/500
 - 0s - loss: 0.0437 - val_loss: 0.0438
Epoch 161/500
 - 0s - loss: 0.0433 - val_loss: 0.0429
Epoch 162/500
 - 0s - loss: 0.0430 - val_loss: 0.0423
Epoch 163/500
 - 0s - loss: 0.0426 - val_loss: 0.0415
Epoch 164/500
 - 0s - loss: 0.0422 - val_loss: 0.0411
Epoch 165/500
 - 0s - loss: 0.0419 - val_loss: 0.0401
Epoch 166/500
 - 0s - loss: 0.0416 - val_loss: 0.0394
Epoch 167/500
 - 0s - loss: 0.0413 - val_loss: 0.0388
Epoch 168/500
 - 0s - loss: 0.0410 - val_loss: 0.0381
Epoch 169/500
 - 0s - loss: 0.0407 - val_loss: 0.0377
Epoch 170/500
 - 0s - loss: 0.0404 - val_loss: 0.0374
Epoch 171/500
 - 0s - loss: 0.0401 - val_loss: 0.0365
Epoch 172/500
 - 0s - loss: 0.0399 - val_loss: 0.0361
Epoch 173/500
 - 0s - loss: 0.0396 - val_loss: 0.0359
Epoch 174/500
 - 0s - loss: 0.0394 - val_loss: 0.0352
Epoch 175/500
 - 0s - loss: 0.0392 - val_loss: 0.0348
Epoch 176/500
 - 0s - loss: 0.0389 - val_loss: 0.0344
Epoch 177/500
 - 0s - loss: 0.0387 - val_loss: 0.0336
Epoch 178/500
 - 0s - loss: 0.0385 - val_loss: 0.0331
Epoch 179/500
 - 0s - loss: 0.0383 - val_loss: 0.0324
Epoch 180/500
 - 0s - loss: 0.0381 - val_loss: 0.0322
Epoch 181/500
 - 0s - loss: 0.0379 - val_loss: 0.0309
Epoch 182/500
 - 0s - loss: 0.0377 - val_loss: 0.0312
Epoch 183/500
 - 0s - loss: 0.0375 - val_loss: 0.0302
Epoch 184/500
 - 0s - loss: 0.0373 - val_loss: 0.0306
Epoch 185/500
 - 0s - loss: 0.0371 - val_loss: 0.0298
Epoch 186/500
 - 0s - loss: 0.0370 - val_loss: 0.0297
Epoch 187/500
 - 0s - loss: 0.0369 - val_loss: 0.0295
Epoch 188/500
 - 0s - loss: 0.0367 - val_loss: 0.0291
Epoch 189/500
 - 0s - loss: 0.0366 - val_loss: 0.0285
Epoch 190/500
 - 0s - loss: 0.0364 - val_loss: 0.0282
Epoch 191/500
 - 0s - loss: 0.0364 - val_loss: 0.0280
Epoch 192/500
 - 0s - loss: 0.0362 - val_loss: 0.0279
Epoch 193/500
 - 0s - loss: 0.0361 - val_loss: 0.0267
Epoch 194/500
 - 0s - loss: 0.0360 - val_loss: 0.0271
Epoch 195/500
 - 0s - loss: 0.0359 - val_loss: 0.0263
Epoch 196/500
 - 0s - loss: 0.0357 - val_loss: 0.0260
Epoch 197/500
 - 0s - loss: 0.0356 - val_loss: 0.0259
Epoch 198/500
 - 0s - loss: 0.0355 - val_loss: 0.0255
Epoch 199/500
 - 0s - loss: 0.0354 - val_loss: 0.0252
Epoch 200/500
 - 0s - loss: 0.0353 - val_loss: 0.0247
Epoch 201/500
 - 0s - loss: 0.0352 - val_loss: 0.0244
Epoch 202/500
 - 0s - loss: 0.0351 - val_loss: 0.0242
Epoch 203/500
 - 0s - loss: 0.0350 - val_loss: 0.0237
Epoch 204/500
 - 0s - loss: 0.0349 - val_loss: 0.0235
Epoch 205/500
 - 0s - loss: 0.0348 - val_loss: 0.0233
Epoch 206/500
 - 0s - loss: 0.0346 - val_loss: 0.0227
Epoch 207/500
 - 0s - loss: 0.0345 - val_loss: 0.0223
Epoch 208/500
 - 0s - loss: 0.0344 - val_loss: 0.0221
Epoch 209/500
 - 0s - loss: 0.0343 - val_loss: 0.0219
Epoch 210/500
 - 0s - loss: 0.0343 - val_loss: 0.0217
Epoch 211/500
 - 0s - loss: 0.0342 - val_loss: 0.0214
Epoch 212/500
 - 0s - loss: 0.0341 - val_loss: 0.0213
Epoch 213/500
 - 0s - loss: 0.0340 - val_loss: 0.0209
Epoch 214/500
 - 0s - loss: 0.0339 - val_loss: 0.0207
Epoch 215/500
 - 0s - loss: 0.0338 - val_loss: 0.0207
Epoch 216/500
 - 0s - loss: 0.0337 - val_loss: 0.0204
Epoch 217/500
 - 0s - loss: 0.0337 - val_loss: 0.0202
Epoch 218/500
 - 0s - loss: 0.0336 - val_loss: 0.0201
Epoch 219/500
 - 0s - loss: 0.0335 - val_loss: 0.0199
Epoch 220/500
 - 0s - loss: 0.0334 - val_loss: 0.0196
Epoch 221/500
 - 0s - loss: 0.0334 - val_loss: 0.0195
Epoch 222/500
 - 0s - loss: 0.0333 - val_loss: 0.0194
Epoch 223/500
 - 0s - loss: 0.0332 - val_loss: 0.0193
Epoch 224/500
 - 0s - loss: 0.0332 - val_loss: 0.0194
Epoch 225/500
 - 0s - loss: 0.0331 - val_loss: 0.0192
Epoch 226/500
 - 0s - loss: 0.0330 - val_loss: 0.0191
Epoch 227/500
 - 0s - loss: 0.0329 - val_loss: 0.0190
Epoch 228/500
 - 0s - loss: 0.0329 - val_loss: 0.0189
Epoch 229/500
 - 0s - loss: 0.0328 - val_loss: 0.0189
Epoch 230/500
 - 0s - loss: 0.0327 - val_loss: 0.0187
Epoch 231/500
 - 0s - loss: 0.0327 - val_loss: 0.0188
Epoch 232/500
 - 0s - loss: 0.0326 - val_loss: 0.0185
Epoch 233/500
 - 0s - loss: 0.0326 - val_loss: 0.0187
Epoch 234/500
 - 0s - loss: 0.0325 - val_loss: 0.0183
Epoch 235/500
 - 0s - loss: 0.0325 - val_loss: 0.0184
Epoch 236/500
 - 0s - loss: 0.0325 - val_loss: 0.0183
Epoch 237/500
 - 0s - loss: 0.0324 - val_loss: 0.0182
Epoch 238/500
 - 0s - loss: 0.0324 - val_loss: 0.0180
Epoch 239/500
 - 0s - loss: 0.0323 - val_loss: 0.0184
Epoch 240/500
 - 0s - loss: 0.0323 - val_loss: 0.0178
Epoch 241/500
 - 0s - loss: 0.0322 - val_loss: 0.0181
Epoch 242/500
 - 0s - loss: 0.0321 - val_loss: 0.0177
Epoch 243/500
 - 0s - loss: 0.0321 - val_loss: 0.0178
Epoch 244/500
 - 0s - loss: 0.0321 - val_loss: 0.0178
Epoch 245/500
 - 0s - loss: 0.0320 - val_loss: 0.0178
Epoch 246/500
 - 0s - loss: 0.0320 - val_loss: 0.0174
Epoch 247/500
 - 0s - loss: 0.0319 - val_loss: 0.0179
Epoch 248/500
 - 0s - loss: 0.0319 - val_loss: 0.0173
Epoch 249/500
 - 0s - loss: 0.0318 - val_loss: 0.0174
Epoch 250/500
 - 0s - loss: 0.0318 - val_loss: 0.0172
Epoch 251/500
 - 0s - loss: 0.0317 - val_loss: 0.0174
Epoch 252/500
 - 0s - loss: 0.0317 - val_loss: 0.0172
Epoch 253/500
 - 0s - loss: 0.0316 - val_loss: 0.0170
Epoch 254/500
 - 0s - loss: 0.0316 - val_loss: 0.0170
Epoch 255/500
 - 0s - loss: 0.0316 - val_loss: 0.0171
Epoch 256/500
 - 0s - loss: 0.0314 - val_loss: 0.0168
Epoch 257/500
 - 0s - loss: 0.0314 - val_loss: 0.0168
Epoch 258/500
 - 0s - loss: 0.0314 - val_loss: 0.0168
Epoch 259/500
 - 0s - loss: 0.0313 - val_loss: 0.0166
Epoch 260/500
 - 0s - loss: 0.0313 - val_loss: 0.0166
Epoch 261/500
 - 0s - loss: 0.0312 - val_loss: 0.0166
Epoch 262/500
 - 0s - loss: 0.0312 - val_loss: 0.0165
Epoch 263/500
 - 0s - loss: 0.0312 - val_loss: 0.0166
Epoch 264/500
 - 0s - loss: 0.0311 - val_loss: 0.0164
Epoch 265/500
 - 0s - loss: 0.0310 - val_loss: 0.0162
Epoch 266/500
 - 0s - loss: 0.0310 - val_loss: 0.0167
Epoch 267/500
 - 0s - loss: 0.0310 - val_loss: 0.0162
Epoch 268/500
 - 0s - loss: 0.0310 - val_loss: 0.0163
Epoch 269/500
 - 0s - loss: 0.0309 - val_loss: 0.0159
Epoch 270/500
 - 0s - loss: 0.0309 - val_loss: 0.0161
Epoch 271/500
 - 0s - loss: 0.0309 - val_loss: 0.0161
Epoch 272/500
 - 0s - loss: 0.0308 - val_loss: 0.0160
Epoch 273/500
 - 0s - loss: 0.0307 - val_loss: 0.0156
Epoch 274/500
 - 0s - loss: 0.0307 - val_loss: 0.0164
Epoch 275/500
 - 0s - loss: 0.0307 - val_loss: 0.0159
Epoch 276/500
 - 0s - loss: 0.0306 - val_loss: 0.0157
Epoch 277/500
 - 0s - loss: 0.0306 - val_loss: 0.0161
Epoch 278/500
 - 0s - loss: 0.0306 - val_loss: 0.0160
Epoch 279/500
 - 0s - loss: 0.0305 - val_loss: 0.0159
Epoch 280/500
 - 0s - loss: 0.0304 - val_loss: 0.0156
Epoch 281/500
 - 0s - loss: 0.0304 - val_loss: 0.0160
Epoch 282/500
 - 0s - loss: 0.0304 - val_loss: 0.0158
Epoch 283/500
 - 0s - loss: 0.0303 - val_loss: 0.0157
Epoch 284/500
 - 0s - loss: 0.0303 - val_loss: 0.0158
Epoch 285/500
 - 0s - loss: 0.0303 - val_loss: 0.0158
Epoch 286/500
 - 0s - loss: 0.0303 - val_loss: 0.0156
Epoch 287/500
 - 0s - loss: 0.0302 - val_loss: 0.0157
Epoch 288/500
 - 0s - loss: 0.0302 - val_loss: 0.0156
Epoch 289/500
 - 0s - loss: 0.0301 - val_loss: 0.0155
Epoch 290/500
 - 0s - loss: 0.0302 - val_loss: 0.0156
Epoch 291/500
 - 0s - loss: 0.0300 - val_loss: 0.0154
Epoch 292/500
 - 0s - loss: 0.0301 - val_loss: 0.0157
Epoch 293/500
 - 0s - loss: 0.0300 - val_loss: 0.0154
Epoch 294/500
 - 0s - loss: 0.0300 - val_loss: 0.0155
Epoch 295/500
 - 0s - loss: 0.0299 - val_loss: 0.0153
Epoch 296/500
 - 0s - loss: 0.0299 - val_loss: 0.0154
Epoch 297/500
 - 0s - loss: 0.0298 - val_loss: 0.0152
Epoch 298/500
 - 0s - loss: 0.0299 - val_loss: 0.0153
Epoch 299/500
 - 0s - loss: 0.0298 - val_loss: 0.0152
Epoch 300/500
 - 0s - loss: 0.0298 - val_loss: 0.0153
Epoch 301/500
 - 0s - loss: 0.0297 - val_loss: 0.0153
Epoch 302/500
 - 0s - loss: 0.0297 - val_loss: 0.0152
Epoch 303/500
 - 0s - loss: 0.0297 - val_loss: 0.0151
Epoch 304/500
 - 0s - loss: 0.0296 - val_loss: 0.0151
Epoch 305/500
 - 0s - loss: 0.0296 - val_loss: 0.0152
Epoch 306/500
 - 0s - loss: 0.0296 - val_loss: 0.0151
Epoch 307/500
 - 0s - loss: 0.0295 - val_loss: 0.0151
Epoch 308/500
 - 0s - loss: 0.0295 - val_loss: 0.0150
Epoch 309/500
 - 0s - loss: 0.0295 - val_loss: 0.0151
Epoch 310/500
 - 0s - loss: 0.0294 - val_loss: 0.0150
Epoch 311/500
 - 0s - loss: 0.0294 - val_loss: 0.0151
Epoch 312/500
 - 0s - loss: 0.0293 - val_loss: 0.0149
Epoch 313/500
 - 0s - loss: 0.0293 - val_loss: 0.0151
Epoch 314/500
 - 0s - loss: 0.0293 - val_loss: 0.0150
Epoch 315/500
 - 0s - loss: 0.0293 - val_loss: 0.0150
Epoch 316/500
 - 0s - loss: 0.0292 - val_loss: 0.0150
Epoch 317/500
 - 0s - loss: 0.0292 - val_loss: 0.0150
Epoch 318/500
 - 0s - loss: 0.0291 - val_loss: 0.0150
Epoch 319/500
 - 0s - loss: 0.0292 - val_loss: 0.0149
Epoch 320/500
 - 0s - loss: 0.0291 - val_loss: 0.0149
Epoch 321/500
 - 0s - loss: 0.0290 - val_loss: 0.0150
Epoch 322/500
 - 0s - loss: 0.0291 - val_loss: 0.0149
Epoch 323/500
 - 0s - loss: 0.0290 - val_loss: 0.0148
Epoch 324/500
 - 0s - loss: 0.0290 - val_loss: 0.0149
Epoch 325/500
 - 0s - loss: 0.0290 - val_loss: 0.0148
Epoch 326/500
 - 0s - loss: 0.0289 - val_loss: 0.0148
Epoch 327/500
 - 0s - loss: 0.0289 - val_loss: 0.0148
Epoch 328/500
 - 0s - loss: 0.0289 - val_loss: 0.0149
Epoch 329/500
 - 0s - loss: 0.0288 - val_loss: 0.0148
Epoch 330/500
 - 0s - loss: 0.0288 - val_loss: 0.0148
Epoch 331/500
 - 0s - loss: 0.0288 - val_loss: 0.0148
Epoch 332/500
 - 0s - loss: 0.0288 - val_loss: 0.0150
Epoch 333/500
 - 0s - loss: 0.0287 - val_loss: 0.0148
Epoch 334/500
 - 0s - loss: 0.0287 - val_loss: 0.0148
Epoch 335/500
 - 0s - loss: 0.0287 - val_loss: 0.0148
Epoch 336/500
 - 0s - loss: 0.0286 - val_loss: 0.0149
Epoch 337/500
 - 0s - loss: 0.0286 - val_loss: 0.0147
Epoch 338/500
 - 0s - loss: 0.0286 - val_loss: 0.0148
Epoch 339/500
 - 0s - loss: 0.0286 - val_loss: 0.0147
Epoch 340/500
 - 0s - loss: 0.0286 - val_loss: 0.0147
Epoch 341/500
 - 0s - loss: 0.0285 - val_loss: 0.0147
Epoch 342/500
 - 0s - loss: 0.0285 - val_loss: 0.0146
Epoch 343/500
 - 0s - loss: 0.0285 - val_loss: 0.0146
Epoch 344/500
 - 0s - loss: 0.0285 - val_loss: 0.0145
Epoch 345/500
 - 0s - loss: 0.0285 - val_loss: 0.0146
Epoch 346/500
 - 0s - loss: 0.0284 - val_loss: 0.0146
Epoch 347/500
 - 0s - loss: 0.0284 - val_loss: 0.0146
Epoch 348/500
 - 0s - loss: 0.0284 - val_loss: 0.0145
Epoch 349/500
 - 0s - loss: 0.0284 - val_loss: 0.0146
Epoch 350/500
 - 0s - loss: 0.0283 - val_loss: 0.0146
Epoch 351/500
 - 0s - loss: 0.0283 - val_loss: 0.0145
Epoch 352/500
 - 0s - loss: 0.0283 - val_loss: 0.0144
Epoch 353/500
 - 0s - loss: 0.0283 - val_loss: 0.0144
Epoch 354/500
 - 0s - loss: 0.0283 - val_loss: 0.0145
Epoch 355/500
 - 0s - loss: 0.0283 - val_loss: 0.0145
Epoch 356/500
 - 0s - loss: 0.0282 - val_loss: 0.0144
Epoch 357/500
 - 0s - loss: 0.0282 - val_loss: 0.0144
Epoch 358/500
 - 0s - loss: 0.0282 - val_loss: 0.0143
Epoch 359/500
 - 0s - loss: 0.0282 - val_loss: 0.0144
Epoch 360/500
 - 0s - loss: 0.0282 - val_loss: 0.0143
Epoch 361/500
 - 0s - loss: 0.0282 - val_loss: 0.0144
Epoch 362/500
 - 0s - loss: 0.0281 - val_loss: 0.0143
Epoch 363/500
 - 0s - loss: 0.0281 - val_loss: 0.0143
Epoch 364/500
 - 0s - loss: 0.0281 - val_loss: 0.0143
Epoch 365/500
 - 0s - loss: 0.0281 - val_loss: 0.0142
Epoch 366/500
 - 0s - loss: 0.0281 - val_loss: 0.0142
Epoch 367/500
 - 0s - loss: 0.0281 - val_loss: 0.0142
Epoch 368/500
 - 0s - loss: 0.0280 - val_loss: 0.0141
Epoch 369/500
 - 0s - loss: 0.0281 - val_loss: 0.0144
Epoch 370/500
 - 0s - loss: 0.0280 - val_loss: 0.0141
Epoch 371/500
 - 0s - loss: 0.0280 - val_loss: 0.0141
Epoch 372/500
 - 0s - loss: 0.0280 - val_loss: 0.0141
Epoch 373/500
 - 0s - loss: 0.0279 - val_loss: 0.0140
Epoch 374/500
 - 0s - loss: 0.0279 - val_loss: 0.0141
Epoch 375/500
 - 0s - loss: 0.0279 - val_loss: 0.0141
Epoch 376/500
 - 0s - loss: 0.0279 - val_loss: 0.0140
Epoch 377/500
 - 0s - loss: 0.0279 - val_loss: 0.0142
Epoch 378/500
 - 0s - loss: 0.0279 - val_loss: 0.0143
Epoch 379/500
 - 0s - loss: 0.0278 - val_loss: 0.0139
Epoch 380/500
 - 0s - loss: 0.0279 - val_loss: 0.0143
Epoch 381/500
 - 0s - loss: 0.0278 - val_loss: 0.0141
Epoch 382/500
 - 0s - loss: 0.0278 - val_loss: 0.0139
Epoch 383/500
 - 0s - loss: 0.0278 - val_loss: 0.0142
Epoch 384/500
 - 0s - loss: 0.0278 - val_loss: 0.0141
Epoch 385/500
 - 0s - loss: 0.0277 - val_loss: 0.0140
Epoch 386/500
 - 0s - loss: 0.0277 - val_loss: 0.0139
Epoch 387/500
 - 0s - loss: 0.0277 - val_loss: 0.0143
Epoch 388/500
 - 0s - loss: 0.0277 - val_loss: 0.0142
Epoch 389/500
 - 0s - loss: 0.0277 - val_loss: 0.0139
Epoch 390/500
 - 0s - loss: 0.0277 - val_loss: 0.0139
Epoch 391/500
 - 0s - loss: 0.0277 - val_loss: 0.0141
Epoch 392/500
 - 0s - loss: 0.0277 - val_loss: 0.0142
Epoch 393/500
 - 0s - loss: 0.0276 - val_loss: 0.0139
Epoch 394/500
 - 0s - loss: 0.0276 - val_loss: 0.0140
Epoch 395/500
 - 0s - loss: 0.0276 - val_loss: 0.0142
Epoch 396/500
 - 0s - loss: 0.0276 - val_loss: 0.0143
Epoch 397/500
 - 0s - loss: 0.0276 - val_loss: 0.0140
Epoch 398/500
 - 0s - loss: 0.0276 - val_loss: 0.0140
Epoch 399/500
 - 0s - loss: 0.0276 - val_loss: 0.0139
Epoch 400/500
 - 0s - loss: 0.0275 - val_loss: 0.0140
Epoch 401/500
 - 0s - loss: 0.0275 - val_loss: 0.0140
Epoch 402/500
 - 0s - loss: 0.0275 - val_loss: 0.0139
Epoch 403/500
 - 0s - loss: 0.0275 - val_loss: 0.0140
Epoch 404/500
 - 0s - loss: 0.0275 - val_loss: 0.0138
Epoch 405/500
 - 0s - loss: 0.0275 - val_loss: 0.0141
Epoch 406/500
 - 0s - loss: 0.0275 - val_loss: 0.0141
Epoch 407/500
 - 0s - loss: 0.0274 - val_loss: 0.0139
Epoch 408/500
 - 0s - loss: 0.0274 - val_loss: 0.0137
Epoch 409/500
 - 0s - loss: 0.0275 - val_loss: 0.0144
Epoch 410/500
 - 0s - loss: 0.0274 - val_loss: 0.0139
Epoch 411/500
 - 0s - loss: 0.0274 - val_loss: 0.0139
Epoch 412/500
 - 0s - loss: 0.0274 - val_loss: 0.0138
Epoch 413/500
 - 0s - loss: 0.0274 - val_loss: 0.0140
Epoch 414/500
 - 0s - loss: 0.0274 - val_loss: 0.0138
Epoch 415/500
 - 0s - loss: 0.0274 - val_loss: 0.0138
Epoch 416/500
 - 0s - loss: 0.0274 - val_loss: 0.0138
Epoch 417/500
 - 0s - loss: 0.0274 - val_loss: 0.0140
Epoch 418/500
 - 0s - loss: 0.0273 - val_loss: 0.0138
Epoch 419/500
 - 0s - loss: 0.0273 - val_loss: 0.0138
Epoch 420/500
 - 0s - loss: 0.0273 - val_loss: 0.0137
Epoch 421/500
 - 0s - loss: 0.0274 - val_loss: 0.0139
Epoch 422/500
 - 0s - loss: 0.0273 - val_loss: 0.0137
Epoch 423/500
 - 0s - loss: 0.0273 - val_loss: 0.0137
Epoch 424/500
 - 0s - loss: 0.0273 - val_loss: 0.0137
Epoch 425/500
 - 0s - loss: 0.0273 - val_loss: 0.0138
Epoch 426/500
 - 0s - loss: 0.0273 - val_loss: 0.0137
Epoch 427/500
 - 0s - loss: 0.0272 - val_loss: 0.0137
Epoch 428/500
 - 0s - loss: 0.0273 - val_loss: 0.0139
Epoch 429/500
 - 0s - loss: 0.0272 - val_loss: 0.0137
Epoch 430/500
 - 0s - loss: 0.0272 - val_loss: 0.0137
Epoch 431/500
 - 0s - loss: 0.0272 - val_loss: 0.0136
Epoch 432/500
 - 0s - loss: 0.0272 - val_loss: 0.0136
Epoch 433/500
 - 0s - loss: 0.0272 - val_loss: 0.0137
Epoch 434/500
 - 0s - loss: 0.0272 - val_loss: 0.0137
Epoch 435/500
 - 0s - loss: 0.0272 - val_loss: 0.0135
Epoch 436/500
 - 0s - loss: 0.0272 - val_loss: 0.0137
Epoch 437/500
 - 0s - loss: 0.0272 - val_loss: 0.0136
Epoch 438/500
 - 0s - loss: 0.0272 - val_loss: 0.0137
Epoch 439/500
 - 0s - loss: 0.0272 - val_loss: 0.0137
Epoch 440/500
 - 0s - loss: 0.0271 - val_loss: 0.0137
Epoch 441/500
 - 0s - loss: 0.0271 - val_loss: 0.0136
Epoch 442/500
 - 0s - loss: 0.0271 - val_loss: 0.0137
Epoch 443/500
 - 0s - loss: 0.0271 - val_loss: 0.0136
Epoch 444/500
 - 0s - loss: 0.0271 - val_loss: 0.0137
Epoch 445/500
 - 0s - loss: 0.0271 - val_loss: 0.0137
Epoch 446/500
 - 0s - loss: 0.0271 - val_loss: 0.0135
Epoch 447/500
 - 0s - loss: 0.0271 - val_loss: 0.0136
Epoch 448/500
 - 0s - loss: 0.0271 - val_loss: 0.0137
Epoch 449/500
 - 0s - loss: 0.0271 - val_loss: 0.0135
Epoch 450/500
 - 0s - loss: 0.0271 - val_loss: 0.0136
Epoch 451/500
 - 0s - loss: 0.0271 - val_loss: 0.0135
Epoch 452/500
 - 0s - loss: 0.0271 - val_loss: 0.0134
Epoch 453/500
 - 0s - loss: 0.0271 - val_loss: 0.0134
Epoch 454/500
 - 0s - loss: 0.0270 - val_loss: 0.0134
Epoch 455/500
 - 0s - loss: 0.0270 - val_loss: 0.0134
Epoch 456/500
 - 0s - loss: 0.0271 - val_loss: 0.0136
Epoch 457/500
 - 0s - loss: 0.0270 - val_loss: 0.0134
Epoch 458/500
 - 0s - loss: 0.0270 - val_loss: 0.0133
Epoch 459/500
 - 0s - loss: 0.0270 - val_loss: 0.0135
Epoch 460/500
 - 0s - loss: 0.0270 - val_loss: 0.0134
Epoch 461/500
 - 0s - loss: 0.0270 - val_loss: 0.0133
Epoch 462/500
 - 0s - loss: 0.0270 - val_loss: 0.0133
Epoch 463/500
 - 0s - loss: 0.0270 - val_loss: 0.0133
Epoch 464/500
 - 0s - loss: 0.0270 - val_loss: 0.0135
Epoch 465/500
 - 0s - loss: 0.0270 - val_loss: 0.0134
Epoch 466/500
 - 0s - loss: 0.0269 - val_loss: 0.0133
Epoch 467/500
 - 0s - loss: 0.0270 - val_loss: 0.0132
Epoch 468/500
 - 0s - loss: 0.0270 - val_loss: 0.0134
Epoch 469/500
 - 0s - loss: 0.0269 - val_loss: 0.0132
Epoch 470/500
 - 0s - loss: 0.0269 - val_loss: 0.0133
Epoch 471/500
 - 0s - loss: 0.0269 - val_loss: 0.0132
Epoch 472/500
 - 0s - loss: 0.0269 - val_loss: 0.0133
Epoch 473/500
 - 0s - loss: 0.0269 - val_loss: 0.0132
Epoch 474/500
 - 0s - loss: 0.0269 - val_loss: 0.0132
Epoch 475/500
 - 0s - loss: 0.0269 - val_loss: 0.0132
Epoch 476/500
 - 0s - loss: 0.0269 - val_loss: 0.0131
Epoch 477/500
 - 0s - loss: 0.0269 - val_loss: 0.0130
Epoch 478/500
 - 0s - loss: 0.0269 - val_loss: 0.0131
Epoch 479/500
 - 0s - loss: 0.0269 - val_loss: 0.0131
Epoch 480/500
 - 0s - loss: 0.0269 - val_loss: 0.0131
Epoch 481/500
 - 0s - loss: 0.0268 - val_loss: 0.0132
Epoch 482/500
 - 0s - loss: 0.0268 - val_loss: 0.0131
Epoch 483/500
 - 0s - loss: 0.0268 - val_loss: 0.0131
Epoch 484/500
 - 0s - loss: 0.0268 - val_loss: 0.0130
Epoch 485/500
 - 0s - loss: 0.0268 - val_loss: 0.0131
Epoch 486/500
 - 0s - loss: 0.0268 - val_loss: 0.0131
Epoch 487/500
 - 0s - loss: 0.0268 - val_loss: 0.0129
Epoch 488/500
 - 0s - loss: 0.0268 - val_loss: 0.0131
Epoch 489/500
 - 0s - loss: 0.0268 - val_loss: 0.0129
Epoch 490/500
 - 0s - loss: 0.0268 - val_loss: 0.0129
Epoch 491/500
 - 0s - loss: 0.0268 - val_loss: 0.0130
Epoch 492/500
 - 0s - loss: 0.0268 - val_loss: 0.0130
Epoch 493/500
 - 0s - loss: 0.0267 - val_loss: 0.0129
Epoch 494/500
 - 0s - loss: 0.0267 - val_loss: 0.0129
Epoch 495/500
 - 0s - loss: 0.0268 - val_loss: 0.0129
Epoch 496/500
 - 0s - loss: 0.0268 - val_loss: 0.0129
Epoch 497/500
 - 0s - loss: 0.0267 - val_loss: 0.0129
Epoch 498/500
 - 0s - loss: 0.0267 - val_loss: 0.0129
Epoch 499/500
 - 0s - loss: 0.0267 - val_loss: 0.0128
Epoch 500/500
 - 0s - loss: 0.0267 - val_loss: 0.0130
In [68]:
pyplot.plot(history['loss'], label='train')
pyplot.plot(history['val_loss'], label='validation')
pyplot.legend()
pyplot.show()
In [69]:
# make a prediction
%load_ext autoreload
%autoreload 2
import models
inv_yhat, inv_y, rmse=models.make_lstm_prediction(validation_X,validation_y,model,scaler)
print('LSTM Model on Validation Data RMSE: %.3f' % rmse)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
LSTM Model on Validation Data RMSE: 9.479
In [70]:
%load_ext autoreload
%autoreload 2
import visuals

visuals.plot_series_to_compare(inv_y,inv_yhat,"Actual Price","Predicted Price", "Actual Price Versus LSTM Predicted Price")
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload

7. Bench Mark Model

In this section we will check our bench mark model. As is proposed in my proposal my bench mark model is a simple linear regressor model.

Load the preprocessed data

In [34]:
from pandas import read_csv
from pandas import datetime
from pandas import DataFrame
from pandas import concat
from matplotlib import pyplot
from sklearn.metrics import mean_squared_error
from math import sqrt


# Create lagged dataset
values = pd.DataFrame(df_weekly["Settle"].values)
df_benchmark  = concat([values.shift(1), values], axis=1)
df_benchmark.columns = ['t', 't+1']
display(df_benchmark.head(5))
t t+1
0 NaN 235.50
1 235.50 228.25
2 228.25 235.50
3 235.50 241.00
4 241.00 253.50
In [35]:
# split into train , validation and test sets
X = df_benchmark.values
train, validation, test = X[1:validation_start], X[validation_start:testing_start],X[testing_start:]
train_bench_X, train_bench_y = train[:,0], train[:,1]
validation_bench_X, validation_bench_y = validation[:,0], validation[:,1]
test_bench_X, test_bench_y = test[:,0], test[:,1]
In [36]:
%load_ext autoreload
%autoreload 2
import models
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
In [37]:
# make a prediction
%load_ext autoreload
%autoreload 2
import models
predictions,rmse=models.make_benchmark_model_prediction(validation_bench_X,validation_bench_y)
print('Benchmark Model on Validation Data RMSE: %.3f' % rmse)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
Benchmark Model on Validation Data RMSE: 8.750
In [38]:
%load_ext autoreload
%autoreload 2
import visuals

visuals.plot_series_to_compare(validation_bench_y,predictions,"Actual Price","Predicted Price", "Actual Price Versus Benchmark Model Predicted Price")
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload

8. Test model on unseen data

Test LSTM model on unseen data

In [71]:
# make a prediction
%load_ext autoreload
%autoreload 2
import models
inv_yhat, inv_y, rmse=models.make_lstm_prediction(test_X,test_y,model,scaler)
print('LSTM Moddel on Test Data RMSE: %.3f' % rmse)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
LSTM Moddel on Test Data RMSE: 12.079

Test Benchmark model on unseen data

In [72]:
# make a prediction
%load_ext autoreload
%autoreload 2
import models
predictions,rmse=models.make_benchmark_model_prediction(test_bench_X,test_bench_y)
print('Benchmark Model on Test Data RMSE: %.3f' % rmse)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
Benchmark Model on Test Data RMSE: 8.293

9. Tune basic LSTM Model

In [41]:
%load_ext autoreload
%autoreload 2
import tune_model
tune_model.tune_memmory_cells(train_X,train_y,validation_X,validation_y)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
>1/5 param=1.000000, loss=0.012129
>2/5 param=1.000000, loss=0.014510
>3/5 param=1.000000, loss=0.011251
>4/5 param=1.000000, loss=0.013165
>5/5 param=1.000000, loss=0.012103
>1/5 param=5.000000, loss=0.011613
>2/5 param=5.000000, loss=0.012066
>3/5 param=5.000000, loss=0.011987
>4/5 param=5.000000, loss=0.012024
>5/5 param=5.000000, loss=0.012577
>1/5 param=10.000000, loss=0.012330
>2/5 param=10.000000, loss=0.013115
>3/5 param=10.000000, loss=0.013052
>4/5 param=10.000000, loss=0.011792
>5/5 param=10.000000, loss=0.013219
>1/5 param=25.000000, loss=0.011451
>2/5 param=25.000000, loss=0.013046
>3/5 param=25.000000, loss=0.011217
>4/5 param=25.000000, loss=0.011381
>5/5 param=25.000000, loss=0.011058
>1/5 param=50.000000, loss=0.012644
>2/5 param=50.000000, loss=0.012646
>3/5 param=50.000000, loss=0.011140
>4/5 param=50.000000, loss=0.013345
>5/5 param=50.000000, loss=0.012515
>1/5 param=100.000000, loss=0.011604
>2/5 param=100.000000, loss=0.015141
>3/5 param=100.000000, loss=0.012493
>4/5 param=100.000000, loss=0.012222
>5/5 param=100.000000, loss=0.013316
>1/5 param=200.000000, loss=0.011432
>2/5 param=200.000000, loss=0.012943
>3/5 param=200.000000, loss=0.010766
>4/5 param=200.000000, loss=0.014985
>5/5 param=200.000000, loss=0.011893
              1         5        10        25        50       100       200
count  5.000000  5.000000  5.000000  5.000000  5.000000  5.000000  5.000000
mean   0.012632  0.012053  0.012702  0.011631  0.012458  0.012955  0.012404
std    0.001250  0.000344  0.000618  0.000806  0.000805  0.001368  0.001646
min    0.011251  0.011613  0.011792  0.011058  0.011140  0.011604  0.010766
25%    0.012103  0.011987  0.012330  0.011217  0.012515  0.012222  0.011432
50%    0.012129  0.012024  0.013052  0.011381  0.012644  0.012493  0.011893
75%    0.013165  0.012066  0.013115  0.011451  0.012646  0.013316  0.012943
max    0.014510  0.012577  0.013219  0.013046  0.013345  0.015141  0.014985
In [42]:
%load_ext autoreload
%autoreload 2
import tune_model
tune_model.tune_batch_size(train_X,train_y,validation_X,validation_y)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
>1/5 param=2.000000, loss=0.017062
>2/5 param=2.000000, loss=0.017288
>3/5 param=2.000000, loss=0.019628
>4/5 param=2.000000, loss=0.017816
>5/5 param=2.000000, loss=0.019510
>1/5 param=4.000000, loss=0.012946
>2/5 param=4.000000, loss=0.013468
>3/5 param=4.000000, loss=0.012065
>4/5 param=4.000000, loss=0.012588
>5/5 param=4.000000, loss=0.013342
>1/5 param=8.000000, loss=0.014913
>2/5 param=8.000000, loss=0.016104
>3/5 param=8.000000, loss=0.015724
>4/5 param=8.000000, loss=0.015701
>5/5 param=8.000000, loss=0.014112
>1/5 param=32.000000, loss=0.011369
>2/5 param=32.000000, loss=0.011775
>3/5 param=32.000000, loss=0.012824
>4/5 param=32.000000, loss=0.012704
>5/5 param=32.000000, loss=0.011133
>1/5 param=64.000000, loss=0.011609
>2/5 param=64.000000, loss=0.011532
>3/5 param=64.000000, loss=0.013435
>4/5 param=64.000000, loss=0.011951
>5/5 param=64.000000, loss=0.012349
>1/5 param=128.000000, loss=0.011928
>2/5 param=128.000000, loss=0.012988
>3/5 param=128.000000, loss=0.011940
>4/5 param=128.000000, loss=0.011974
>5/5 param=128.000000, loss=0.011488
>1/5 param=256.000000, loss=0.011780
>2/5 param=256.000000, loss=0.013215
>3/5 param=256.000000, loss=0.012355
>4/5 param=256.000000, loss=0.011390
>5/5 param=256.000000, loss=0.011595
              2         4         8        32        64       128       256
count  5.000000  5.000000  5.000000  5.000000  5.000000  5.000000  5.000000
mean   0.018261  0.012882  0.015311  0.011961  0.012175  0.012064  0.012067
std    0.001226  0.000573  0.000798  0.000769  0.000775  0.000554  0.000736
min    0.017062  0.012065  0.014112  0.011133  0.011532  0.011488  0.011390
25%    0.017288  0.012588  0.014913  0.011369  0.011609  0.011928  0.011595
50%    0.017816  0.012946  0.015701  0.011775  0.011951  0.011940  0.011780
75%    0.019510  0.013342  0.015724  0.012704  0.012349  0.011974  0.012355
max    0.019628  0.013468  0.016104  0.012824  0.013435  0.012988  0.013215
In [85]:
%load_ext autoreload
%autoreload 2
import tune_model
tune_model.tune_learning_rate(train_X,train_y,validation_X,validation_y)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
>1/5 param=0.100000, loss=0.011195
>2/5 param=0.100000, loss=0.011809
>3/5 param=0.100000, loss=0.018709
>4/5 param=0.100000, loss=0.032624
>5/5 param=0.100000, loss=0.015232
>1/5 param=0.001000, loss=0.011950
>2/5 param=0.001000, loss=0.012022
>3/5 param=0.001000, loss=0.012852
>4/5 param=0.001000, loss=0.012499
>5/5 param=0.001000, loss=0.011699
>1/5 param=0.000100, loss=0.033569
>2/5 param=0.000100, loss=0.027969
>3/5 param=0.000100, loss=0.051960
>4/5 param=0.000100, loss=0.043928
>5/5 param=0.000100, loss=0.035140
            0.1     0.001    0.0001
count  5.000000  5.000000  5.000000
mean   0.017914  0.012204  0.038513
std    0.008756  0.000464  0.009449
min    0.011195  0.011699  0.027969
25%    0.011809  0.011950  0.033569
50%    0.015232  0.012022  0.035140
75%    0.018709  0.012499  0.043928
max    0.032624  0.012852  0.051960
In [86]:
%load_ext autoreload
%autoreload 2
import tune_model
tune_model.tune_weight_regularization(train_X,train_y,validation_X,validation_y)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
>1/5 param=1.000000, loss=0.017913
>2/5 param=1.000000, loss=0.017820
>3/5 param=1.000000, loss=0.017644
>4/5 param=1.000000, loss=0.018630
>5/5 param=1.000000, loss=0.018884
>1/5 param=2.000000, loss=0.033898
>2/5 param=2.000000, loss=0.035481
>3/5 param=2.000000, loss=0.036565
>4/5 param=2.000000, loss=0.036193
>5/5 param=2.000000, loss=0.035109
>1/5 param=3.000000, loss=0.012555
>2/5 param=3.000000, loss=0.011829
>3/5 param=3.000000, loss=0.012490
>4/5 param=3.000000, loss=0.011938
>5/5 param=3.000000, loss=0.011889
>1/5 param=4.000000, loss=0.037587
>2/5 param=4.000000, loss=0.039391
>3/5 param=4.000000, loss=0.038415
>4/5 param=4.000000, loss=0.039397
>5/5 param=4.000000, loss=0.038773
              1         2         3         4
count  5.000000  5.000000  5.000000  5.000000
mean   0.018178  0.035449  0.012140  0.038713
std    0.000544  0.001040  0.000352  0.000756
min    0.017644  0.033898  0.011829  0.037587
25%    0.017820  0.035109  0.011889  0.038415
50%    0.017913  0.035481  0.011938  0.038773
75%    0.018630  0.036193  0.012490  0.039391
max    0.018884  0.036565  0.012555  0.039397

10. Test Improved LSTM Model

In [173]:
%load_ext autoreload
%autoreload 2
import models
model,history=models.improved_lstm_model(train_X,train_y,validation_X,validation_y)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
Train on 550 samples, validate on 53 samples
Epoch 1/500
 - 28s - loss: 1.1178 - val_loss: 0.8899
Epoch 2/500
 - 0s - loss: 1.0660 - val_loss: 0.8304
Epoch 3/500
 - 0s - loss: 1.0112 - val_loss: 0.7748
Epoch 4/500
 - 0s - loss: 0.9573 - val_loss: 0.7582
Epoch 5/500
 - 0s - loss: 0.9143 - val_loss: 0.7839
Epoch 6/500
 - 0s - loss: 0.8879 - val_loss: 0.8102
Epoch 7/500
 - 0s - loss: 0.8721 - val_loss: 0.8232
Epoch 8/500
 - 0s - loss: 0.8608 - val_loss: 0.8247
Epoch 9/500
 - 0s - loss: 0.8509 - val_loss: 0.8184
Epoch 10/500
 - 0s - loss: 0.8417 - val_loss: 0.8072
Epoch 11/500
 - 0s - loss: 0.8328 - val_loss: 0.7936
Epoch 12/500
 - 0s - loss: 0.8241 - val_loss: 0.7790
Epoch 13/500
 - 0s - loss: 0.8158 - val_loss: 0.7646
Epoch 14/500
 - 0s - loss: 0.8076 - val_loss: 0.7510
Epoch 15/500
 - 0s - loss: 0.7994 - val_loss: 0.7384
Epoch 16/500
 - 0s - loss: 0.7911 - val_loss: 0.7266
Epoch 17/500
 - 0s - loss: 0.7828 - val_loss: 0.7157
Epoch 18/500
 - 0s - loss: 0.7745 - val_loss: 0.7062
Epoch 19/500
 - 0s - loss: 0.7660 - val_loss: 0.6976
Epoch 20/500
 - 0s - loss: 0.7574 - val_loss: 0.6894
Epoch 21/500
 - 0s - loss: 0.7487 - val_loss: 0.6814
Epoch 22/500
 - 0s - loss: 0.7401 - val_loss: 0.6738
Epoch 23/500
 - 0s - loss: 0.7314 - val_loss: 0.6663
Epoch 24/500
 - 0s - loss: 0.7227 - val_loss: 0.6589
Epoch 25/500
 - 0s - loss: 0.7139 - val_loss: 0.6515
Epoch 26/500
 - 0s - loss: 0.7052 - val_loss: 0.6437
Epoch 27/500
 - 0s - loss: 0.6966 - val_loss: 0.6356
Epoch 28/500
 - 0s - loss: 0.6880 - val_loss: 0.6271
Epoch 29/500
 - 0s - loss: 0.6795 - val_loss: 0.6186
Epoch 30/500
 - 0s - loss: 0.6711 - val_loss: 0.6100
Epoch 31/500
 - 0s - loss: 0.6626 - val_loss: 0.6013
Epoch 32/500
 - 0s - loss: 0.6541 - val_loss: 0.5929
Epoch 33/500
 - 0s - loss: 0.6456 - val_loss: 0.5849
Epoch 34/500
 - 0s - loss: 0.6370 - val_loss: 0.5772
Epoch 35/500
 - 0s - loss: 0.6283 - val_loss: 0.5696
Epoch 36/500
 - 0s - loss: 0.6196 - val_loss: 0.5621
Epoch 37/500
 - 0s - loss: 0.6109 - val_loss: 0.5544
Epoch 38/500
 - 0s - loss: 0.6021 - val_loss: 0.5465
Epoch 39/500
 - 0s - loss: 0.5934 - val_loss: 0.5383
Epoch 40/500
 - 0s - loss: 0.5847 - val_loss: 0.5299
Epoch 41/500
 - 0s - loss: 0.5760 - val_loss: 0.5214
Epoch 42/500
 - 0s - loss: 0.5672 - val_loss: 0.5134
Epoch 43/500
 - 0s - loss: 0.5584 - val_loss: 0.5058
Epoch 44/500
 - 0s - loss: 0.5495 - val_loss: 0.4984
Epoch 45/500
 - 0s - loss: 0.5405 - val_loss: 0.4913
Epoch 46/500
 - 0s - loss: 0.5318 - val_loss: 0.4844
Epoch 47/500
 - 0s - loss: 0.5231 - val_loss: 0.4776
Epoch 48/500
 - 0s - loss: 0.5147 - val_loss: 0.4704
Epoch 49/500
 - 0s - loss: 0.5064 - val_loss: 0.4633
Epoch 50/500
 - 0s - loss: 0.4984 - val_loss: 0.4568
Epoch 51/500
 - 0s - loss: 0.4907 - val_loss: 0.4517
Epoch 52/500
 - 0s - loss: 0.4830 - val_loss: 0.4471
Epoch 53/500
 - 0s - loss: 0.4754 - val_loss: 0.4425
Epoch 54/500
 - 0s - loss: 0.4682 - val_loss: 0.4378
Epoch 55/500
 - 0s - loss: 0.4612 - val_loss: 0.4334
Epoch 56/500
 - 0s - loss: 0.4543 - val_loss: 0.4288
Epoch 57/500
 - 0s - loss: 0.4477 - val_loss: 0.4245
Epoch 58/500
 - 0s - loss: 0.4412 - val_loss: 0.4199
Epoch 59/500
 - 0s - loss: 0.4349 - val_loss: 0.4151
Epoch 60/500
 - 0s - loss: 0.4288 - val_loss: 0.4105
Epoch 61/500
 - 0s - loss: 0.4229 - val_loss: 0.4059
Epoch 62/500
 - 0s - loss: 0.4171 - val_loss: 0.4007
Epoch 63/500
 - 0s - loss: 0.4115 - val_loss: 0.3955
Epoch 64/500
 - 0s - loss: 0.4062 - val_loss: 0.3905
Epoch 65/500
 - 0s - loss: 0.4011 - val_loss: 0.3859
Epoch 66/500
 - 0s - loss: 0.3960 - val_loss: 0.3811
Epoch 67/500
 - 0s - loss: 0.3910 - val_loss: 0.3763
Epoch 68/500
 - 0s - loss: 0.3862 - val_loss: 0.3716
Epoch 69/500
 - 0s - loss: 0.3815 - val_loss: 0.3670
Epoch 70/500
 - 0s - loss: 0.3769 - val_loss: 0.3622
Epoch 71/500
 - 0s - loss: 0.3724 - val_loss: 0.3574
Epoch 72/500
 - 0s - loss: 0.3679 - val_loss: 0.3528
Epoch 73/500
 - 0s - loss: 0.3636 - val_loss: 0.3482
Epoch 74/500
 - 0s - loss: 0.3593 - val_loss: 0.3435
Epoch 75/500
 - 0s - loss: 0.3551 - val_loss: 0.3392
Epoch 76/500
 - 0s - loss: 0.3509 - val_loss: 0.3352
Epoch 77/500
 - 0s - loss: 0.3468 - val_loss: 0.3307
Epoch 78/500
 - 0s - loss: 0.3427 - val_loss: 0.3266
Epoch 79/500
 - 0s - loss: 0.3387 - val_loss: 0.3226
Epoch 80/500
 - 0s - loss: 0.3348 - val_loss: 0.3184
Epoch 81/500
 - 0s - loss: 0.3309 - val_loss: 0.3144
Epoch 82/500
 - 0s - loss: 0.3270 - val_loss: 0.3106
Epoch 83/500
 - 0s - loss: 0.3232 - val_loss: 0.3065
Epoch 84/500
 - 0s - loss: 0.3194 - val_loss: 0.3026
Epoch 85/500
 - 0s - loss: 0.3157 - val_loss: 0.2989
Epoch 86/500
 - 0s - loss: 0.3120 - val_loss: 0.2952
Epoch 87/500
 - 0s - loss: 0.3083 - val_loss: 0.2913
Epoch 88/500
 - 0s - loss: 0.3047 - val_loss: 0.2876
Epoch 89/500
 - 0s - loss: 0.3012 - val_loss: 0.2841
Epoch 90/500
 - 0s - loss: 0.2976 - val_loss: 0.2805
Epoch 91/500
 - 0s - loss: 0.2942 - val_loss: 0.2770
Epoch 92/500
 - 0s - loss: 0.2907 - val_loss: 0.2733
Epoch 93/500
 - 0s - loss: 0.2873 - val_loss: 0.2701
Epoch 94/500
 - 0s - loss: 0.2839 - val_loss: 0.2666
Epoch 95/500
 - 0s - loss: 0.2805 - val_loss: 0.2633
Epoch 96/500
 - 0s - loss: 0.2772 - val_loss: 0.2599
Epoch 97/500
 - 0s - loss: 0.2740 - val_loss: 0.2564
Epoch 98/500
 - 0s - loss: 0.2707 - val_loss: 0.2535
Epoch 99/500
 - 0s - loss: 0.2675 - val_loss: 0.2501
Epoch 100/500
 - 0s - loss: 0.2643 - val_loss: 0.2470
Epoch 101/500
 - 0s - loss: 0.2611 - val_loss: 0.2438
Epoch 102/500
 - 0s - loss: 0.2580 - val_loss: 0.2407
Epoch 103/500
 - 0s - loss: 0.2549 - val_loss: 0.2376
Epoch 104/500
 - 0s - loss: 0.2519 - val_loss: 0.2347
Epoch 105/500
 - 0s - loss: 0.2489 - val_loss: 0.2315
Epoch 106/500
 - 0s - loss: 0.2459 - val_loss: 0.2287
Epoch 107/500
 - 0s - loss: 0.2429 - val_loss: 0.2256
Epoch 108/500
 - 0s - loss: 0.2400 - val_loss: 0.2227
Epoch 109/500
 - 0s - loss: 0.2371 - val_loss: 0.2199
Epoch 110/500
 - 0s - loss: 0.2342 - val_loss: 0.2170
Epoch 111/500
 - 0s - loss: 0.2314 - val_loss: 0.2143
Epoch 112/500
 - 0s - loss: 0.2286 - val_loss: 0.2114
Epoch 113/500
 - 0s - loss: 0.2258 - val_loss: 0.2087
Epoch 114/500
 - 0s - loss: 0.2231 - val_loss: 0.2060
Epoch 115/500
 - 0s - loss: 0.2204 - val_loss: 0.2033
Epoch 116/500
 - 0s - loss: 0.2177 - val_loss: 0.2008
Epoch 117/500
 - 0s - loss: 0.2150 - val_loss: 0.1981
Epoch 118/500
 - 0s - loss: 0.2124 - val_loss: 0.1954
Epoch 119/500
 - 0s - loss: 0.2098 - val_loss: 0.1930
Epoch 120/500
 - 0s - loss: 0.2073 - val_loss: 0.1903
Epoch 121/500
 - 0s - loss: 0.2047 - val_loss: 0.1878
Epoch 122/500
 - 0s - loss: 0.2022 - val_loss: 0.1854
Epoch 123/500
 - 0s - loss: 0.1997 - val_loss: 0.1829
Epoch 124/500
 - 0s - loss: 0.1973 - val_loss: 0.1805
Epoch 125/500
 - 0s - loss: 0.1948 - val_loss: 0.1781
Epoch 126/500
 - 0s - loss: 0.1924 - val_loss: 0.1758
Epoch 127/500
 - 0s - loss: 0.1901 - val_loss: 0.1735
Epoch 128/500
 - 0s - loss: 0.1877 - val_loss: 0.1711
Epoch 129/500
 - 0s - loss: 0.1854 - val_loss: 0.1690
Epoch 130/500
 - 0s - loss: 0.1831 - val_loss: 0.1666
Epoch 131/500
 - 0s - loss: 0.1808 - val_loss: 0.1642
Epoch 132/500
 - 0s - loss: 0.1786 - val_loss: 0.1622
Epoch 133/500
 - 0s - loss: 0.1764 - val_loss: 0.1599
Epoch 134/500
 - 0s - loss: 0.1742 - val_loss: 0.1578
Epoch 135/500
 - 0s - loss: 0.1720 - val_loss: 0.1557
Epoch 136/500
 - 0s - loss: 0.1699 - val_loss: 0.1535
Epoch 137/500
 - 0s - loss: 0.1678 - val_loss: 0.1515
Epoch 138/500
 - 0s - loss: 0.1657 - val_loss: 0.1493
Epoch 139/500
 - 0s - loss: 0.1636 - val_loss: 0.1475
Epoch 140/500
 - 0s - loss: 0.1616 - val_loss: 0.1454
Epoch 141/500
 - 0s - loss: 0.1595 - val_loss: 0.1434
Epoch 142/500
 - 0s - loss: 0.1575 - val_loss: 0.1415
Epoch 143/500
 - 0s - loss: 0.1556 - val_loss: 0.1395
Epoch 144/500
 - 0s - loss: 0.1536 - val_loss: 0.1377
Epoch 145/500
 - 0s - loss: 0.1517 - val_loss: 0.1357
Epoch 146/500
 - 0s - loss: 0.1498 - val_loss: 0.1340
Epoch 147/500
 - 0s - loss: 0.1479 - val_loss: 0.1320
Epoch 148/500
 - 0s - loss: 0.1461 - val_loss: 0.1301
Epoch 149/500
 - 0s - loss: 0.1442 - val_loss: 0.1285
Epoch 150/500
 - 0s - loss: 0.1424 - val_loss: 0.1265
Epoch 151/500
 - 0s - loss: 0.1406 - val_loss: 0.1250
Epoch 152/500
 - 0s - loss: 0.1389 - val_loss: 0.1230
Epoch 153/500
 - 0s - loss: 0.1371 - val_loss: 0.1215
Epoch 154/500
 - 0s - loss: 0.1354 - val_loss: 0.1197
Epoch 155/500
 - 0s - loss: 0.1337 - val_loss: 0.1181
Epoch 156/500
 - 0s - loss: 0.1320 - val_loss: 0.1163
Epoch 157/500
 - 0s - loss: 0.1304 - val_loss: 0.1148
Epoch 158/500
 - 0s - loss: 0.1287 - val_loss: 0.1132
Epoch 159/500
 - 0s - loss: 0.1271 - val_loss: 0.1115
Epoch 160/500
 - 0s - loss: 0.1255 - val_loss: 0.1100
Epoch 161/500
 - 0s - loss: 0.1239 - val_loss: 0.1084
Epoch 162/500
 - 0s - loss: 0.1224 - val_loss: 0.1069
Epoch 163/500
 - 0s - loss: 0.1209 - val_loss: 0.1054
Epoch 164/500
 - 0s - loss: 0.1193 - val_loss: 0.1039
Epoch 165/500
 - 0s - loss: 0.1178 - val_loss: 0.1023
Epoch 166/500
 - 0s - loss: 0.1164 - val_loss: 0.1010
Epoch 167/500
 - 0s - loss: 0.1149 - val_loss: 0.0995
Epoch 168/500
 - 0s - loss: 0.1135 - val_loss: 0.0981
Epoch 169/500
 - 0s - loss: 0.1120 - val_loss: 0.0968
Epoch 170/500
 - 0s - loss: 0.1107 - val_loss: 0.0953
Epoch 171/500
 - 0s - loss: 0.1093 - val_loss: 0.0941
Epoch 172/500
 - 0s - loss: 0.1079 - val_loss: 0.0926
Epoch 173/500
 - 0s - loss: 0.1065 - val_loss: 0.0913
Epoch 174/500
 - 0s - loss: 0.1052 - val_loss: 0.0900
Epoch 175/500
 - 0s - loss: 0.1039 - val_loss: 0.0887
Epoch 176/500
 - 0s - loss: 0.1026 - val_loss: 0.0874
Epoch 177/500
 - 0s - loss: 0.1013 - val_loss: 0.0863
Epoch 178/500
 - 0s - loss: 0.1001 - val_loss: 0.0848
Epoch 179/500
 - 0s - loss: 0.0988 - val_loss: 0.0838
Epoch 180/500
 - 0s - loss: 0.0976 - val_loss: 0.0824
Epoch 181/500
 - 0s - loss: 0.0964 - val_loss: 0.0814
Epoch 182/500
 - 0s - loss: 0.0952 - val_loss: 0.0801
Epoch 183/500
 - 0s - loss: 0.0940 - val_loss: 0.0792
Epoch 184/500
 - 0s - loss: 0.0929 - val_loss: 0.0778
Epoch 185/500
 - 0s - loss: 0.0917 - val_loss: 0.0769
Epoch 186/500
 - 0s - loss: 0.0906 - val_loss: 0.0756
Epoch 187/500
 - 0s - loss: 0.0895 - val_loss: 0.0746
Epoch 188/500
 - 0s - loss: 0.0884 - val_loss: 0.0734
Epoch 189/500
 - 0s - loss: 0.0873 - val_loss: 0.0726
Epoch 190/500
 - 0s - loss: 0.0863 - val_loss: 0.0712
Epoch 191/500
 - 0s - loss: 0.0852 - val_loss: 0.0704
Epoch 192/500
 - 0s - loss: 0.0842 - val_loss: 0.0693
Epoch 193/500
 - 0s - loss: 0.0832 - val_loss: 0.0681
Epoch 194/500
 - 0s - loss: 0.0822 - val_loss: 0.0673
Epoch 195/500
 - 0s - loss: 0.0812 - val_loss: 0.0662
Epoch 196/500
 - 0s - loss: 0.0802 - val_loss: 0.0655
Epoch 197/500
 - 0s - loss: 0.0792 - val_loss: 0.0644
Epoch 198/500
 - 0s - loss: 0.0783 - val_loss: 0.0636
Epoch 199/500
 - 0s - loss: 0.0773 - val_loss: 0.0626
Epoch 200/500
 - 0s - loss: 0.0764 - val_loss: 0.0619
Epoch 201/500
 - 0s - loss: 0.0755 - val_loss: 0.0607
Epoch 202/500
 - 0s - loss: 0.0746 - val_loss: 0.0599
Epoch 203/500
 - 0s - loss: 0.0737 - val_loss: 0.0590
Epoch 204/500
 - 0s - loss: 0.0729 - val_loss: 0.0579
Epoch 205/500
 - 0s - loss: 0.0720 - val_loss: 0.0574
Epoch 206/500
 - 0s - loss: 0.0712 - val_loss: 0.0565
Epoch 207/500
 - 0s - loss: 0.0703 - val_loss: 0.0557
Epoch 208/500
 - 0s - loss: 0.0695 - val_loss: 0.0549
Epoch 209/500
 - 0s - loss: 0.0687 - val_loss: 0.0541
Epoch 210/500
 - 0s - loss: 0.0679 - val_loss: 0.0536
Epoch 211/500
 - 0s - loss: 0.0672 - val_loss: 0.0524
Epoch 212/500
 - 0s - loss: 0.0664 - val_loss: 0.0519
Epoch 213/500
 - 0s - loss: 0.0656 - val_loss: 0.0511
Epoch 214/500
 - 0s - loss: 0.0649 - val_loss: 0.0504
Epoch 215/500
 - 0s - loss: 0.0642 - val_loss: 0.0497
Epoch 216/500
 - 0s - loss: 0.0634 - val_loss: 0.0489
Epoch 217/500
 - 0s - loss: 0.0627 - val_loss: 0.0484
Epoch 218/500
 - 0s - loss: 0.0620 - val_loss: 0.0474
Epoch 219/500
 - 0s - loss: 0.0613 - val_loss: 0.0469
Epoch 220/500
 - 0s - loss: 0.0607 - val_loss: 0.0461
Epoch 221/500
 - 0s - loss: 0.0600 - val_loss: 0.0456
Epoch 222/500
 - 0s - loss: 0.0593 - val_loss: 0.0449
Epoch 223/500
 - 0s - loss: 0.0587 - val_loss: 0.0442
Epoch 224/500
 - 0s - loss: 0.0580 - val_loss: 0.0437
Epoch 225/500
 - 0s - loss: 0.0574 - val_loss: 0.0430
Epoch 226/500
 - 0s - loss: 0.0568 - val_loss: 0.0424
Epoch 227/500
 - 0s - loss: 0.0562 - val_loss: 0.0418
Epoch 228/500
 - 0s - loss: 0.0556 - val_loss: 0.0412
Epoch 229/500
 - 0s - loss: 0.0550 - val_loss: 0.0407
Epoch 230/500
 - 0s - loss: 0.0544 - val_loss: 0.0400
Epoch 231/500
 - 0s - loss: 0.0539 - val_loss: 0.0396
Epoch 232/500
 - 0s - loss: 0.0533 - val_loss: 0.0390
Epoch 233/500
 - 0s - loss: 0.0528 - val_loss: 0.0384
Epoch 234/500
 - 0s - loss: 0.0522 - val_loss: 0.0382
Epoch 235/500
 - 0s - loss: 0.0517 - val_loss: 0.0371
Epoch 236/500
 - 0s - loss: 0.0512 - val_loss: 0.0370
Epoch 237/500
 - 0s - loss: 0.0507 - val_loss: 0.0362
Epoch 238/500
 - 0s - loss: 0.0502 - val_loss: 0.0360
Epoch 239/500
 - 0s - loss: 0.0497 - val_loss: 0.0353
Epoch 240/500
 - 0s - loss: 0.0492 - val_loss: 0.0351
Epoch 241/500
 - 0s - loss: 0.0487 - val_loss: 0.0345
Epoch 242/500
 - 0s - loss: 0.0482 - val_loss: 0.0341
Epoch 243/500
 - 0s - loss: 0.0478 - val_loss: 0.0338
Epoch 244/500
 - 0s - loss: 0.0473 - val_loss: 0.0330
Epoch 245/500
 - 0s - loss: 0.0469 - val_loss: 0.0330
Epoch 246/500
 - 0s - loss: 0.0465 - val_loss: 0.0321
Epoch 247/500
 - 0s - loss: 0.0460 - val_loss: 0.0321
Epoch 248/500
 - 0s - loss: 0.0456 - val_loss: 0.0314
Epoch 249/500
 - 0s - loss: 0.0452 - val_loss: 0.0313
Epoch 250/500
 - 0s - loss: 0.0448 - val_loss: 0.0306
Epoch 251/500
 - 0s - loss: 0.0444 - val_loss: 0.0304
Epoch 252/500
 - 0s - loss: 0.0440 - val_loss: 0.0298
Epoch 253/500
 - 0s - loss: 0.0436 - val_loss: 0.0295
Epoch 254/500
 - 0s - loss: 0.0432 - val_loss: 0.0291
Epoch 255/500
 - 0s - loss: 0.0428 - val_loss: 0.0287
Epoch 256/500
 - 0s - loss: 0.0424 - val_loss: 0.0284
Epoch 257/500
 - 0s - loss: 0.0421 - val_loss: 0.0280
Epoch 258/500
 - 0s - loss: 0.0417 - val_loss: 0.0276
Epoch 259/500
 - 0s - loss: 0.0414 - val_loss: 0.0273
Epoch 260/500
 - 0s - loss: 0.0410 - val_loss: 0.0270
Epoch 261/500
 - 0s - loss: 0.0407 - val_loss: 0.0267
Epoch 262/500
 - 0s - loss: 0.0404 - val_loss: 0.0263
Epoch 263/500
 - 0s - loss: 0.0401 - val_loss: 0.0260
Epoch 264/500
 - 0s - loss: 0.0397 - val_loss: 0.0257
Epoch 265/500
 - 0s - loss: 0.0394 - val_loss: 0.0255
Epoch 266/500
 - 0s - loss: 0.0391 - val_loss: 0.0252
Epoch 267/500
 - 0s - loss: 0.0388 - val_loss: 0.0248
Epoch 268/500
 - 0s - loss: 0.0385 - val_loss: 0.0248
Epoch 269/500
 - 0s - loss: 0.0383 - val_loss: 0.0242
Epoch 270/500
 - 0s - loss: 0.0380 - val_loss: 0.0243
Epoch 271/500
 - 0s - loss: 0.0377 - val_loss: 0.0236
Epoch 272/500
 - 0s - loss: 0.0374 - val_loss: 0.0237
Epoch 273/500
 - 0s - loss: 0.0372 - val_loss: 0.0231
Epoch 274/500
 - 0s - loss: 0.0369 - val_loss: 0.0230
Epoch 275/500
 - 0s - loss: 0.0366 - val_loss: 0.0227
Epoch 276/500
 - 0s - loss: 0.0364 - val_loss: 0.0225
Epoch 277/500
 - 0s - loss: 0.0361 - val_loss: 0.0224
Epoch 278/500
 - 0s - loss: 0.0359 - val_loss: 0.0220
Epoch 279/500
 - 0s - loss: 0.0357 - val_loss: 0.0217
Epoch 280/500
 - 0s - loss: 0.0354 - val_loss: 0.0217
Epoch 281/500
 - 0s - loss: 0.0352 - val_loss: 0.0213
Epoch 282/500
 - 0s - loss: 0.0350 - val_loss: 0.0213
Epoch 283/500
 - 0s - loss: 0.0348 - val_loss: 0.0211
Epoch 284/500
 - 0s - loss: 0.0346 - val_loss: 0.0207
Epoch 285/500
 - 0s - loss: 0.0343 - val_loss: 0.0208
Epoch 286/500
 - 0s - loss: 0.0341 - val_loss: 0.0204
Epoch 287/500
 - 0s - loss: 0.0339 - val_loss: 0.0202
Epoch 288/500
 - 0s - loss: 0.0337 - val_loss: 0.0201
Epoch 289/500
 - 0s - loss: 0.0335 - val_loss: 0.0198
Epoch 290/500
 - 0s - loss: 0.0333 - val_loss: 0.0197
Epoch 291/500
 - 0s - loss: 0.0332 - val_loss: 0.0196
Epoch 292/500
 - 0s - loss: 0.0330 - val_loss: 0.0190
Epoch 293/500
 - 0s - loss: 0.0329 - val_loss: 0.0192
Epoch 294/500
 - 0s - loss: 0.0327 - val_loss: 0.0186
Epoch 295/500
 - 0s - loss: 0.0325 - val_loss: 0.0190
Epoch 296/500
 - 0s - loss: 0.0323 - val_loss: 0.0183
Epoch 297/500
 - 0s - loss: 0.0321 - val_loss: 0.0186
Epoch 298/500
 - 0s - loss: 0.0320 - val_loss: 0.0181
Epoch 299/500
 - 0s - loss: 0.0318 - val_loss: 0.0185
Epoch 300/500
 - 0s - loss: 0.0317 - val_loss: 0.0177
Epoch 301/500
 - 0s - loss: 0.0315 - val_loss: 0.0180
Epoch 302/500
 - 0s - loss: 0.0314 - val_loss: 0.0175
Epoch 303/500
 - 0s - loss: 0.0312 - val_loss: 0.0174
Epoch 304/500
 - 0s - loss: 0.0311 - val_loss: 0.0173
Epoch 305/500
 - 0s - loss: 0.0309 - val_loss: 0.0174
Epoch 306/500
 - 0s - loss: 0.0308 - val_loss: 0.0172
Epoch 307/500
 - 0s - loss: 0.0307 - val_loss: 0.0172
Epoch 308/500
 - 0s - loss: 0.0306 - val_loss: 0.0169
Epoch 309/500
 - 0s - loss: 0.0304 - val_loss: 0.0169
Epoch 310/500
 - 0s - loss: 0.0303 - val_loss: 0.0167
Epoch 311/500
 - 0s - loss: 0.0302 - val_loss: 0.0164
Epoch 312/500
 - 0s - loss: 0.0300 - val_loss: 0.0165
Epoch 313/500
 - 0s - loss: 0.0299 - val_loss: 0.0165
Epoch 314/500
 - 0s - loss: 0.0298 - val_loss: 0.0161
Epoch 315/500
 - 0s - loss: 0.0297 - val_loss: 0.0161
Epoch 316/500
 - 0s - loss: 0.0296 - val_loss: 0.0159
Epoch 317/500
 - 0s - loss: 0.0295 - val_loss: 0.0158
Epoch 318/500
 - 0s - loss: 0.0294 - val_loss: 0.0159
Epoch 319/500
 - 0s - loss: 0.0293 - val_loss: 0.0156
Epoch 320/500
 - 0s - loss: 0.0292 - val_loss: 0.0158
Epoch 321/500
 - 0s - loss: 0.0291 - val_loss: 0.0155
Epoch 322/500
 - 0s - loss: 0.0290 - val_loss: 0.0154
Epoch 323/500
 - 0s - loss: 0.0289 - val_loss: 0.0155
Epoch 324/500
 - 0s - loss: 0.0288 - val_loss: 0.0152
Epoch 325/500
 - 0s - loss: 0.0287 - val_loss: 0.0153
Epoch 326/500
 - 0s - loss: 0.0286 - val_loss: 0.0152
Epoch 327/500
 - 0s - loss: 0.0286 - val_loss: 0.0149
Epoch 328/500
 - 0s - loss: 0.0285 - val_loss: 0.0149
Epoch 329/500
 - 0s - loss: 0.0284 - val_loss: 0.0152
Epoch 330/500
 - 0s - loss: 0.0284 - val_loss: 0.0145
Epoch 331/500
 - 0s - loss: 0.0282 - val_loss: 0.0148
Epoch 332/500
 - 0s - loss: 0.0282 - val_loss: 0.0146
Epoch 333/500
 - 0s - loss: 0.0281 - val_loss: 0.0144
Epoch 334/500
 - 0s - loss: 0.0280 - val_loss: 0.0146
Epoch 335/500
 - 0s - loss: 0.0280 - val_loss: 0.0143
Epoch 336/500
 - 0s - loss: 0.0279 - val_loss: 0.0145
Epoch 337/500
 - 0s - loss: 0.0278 - val_loss: 0.0142
Epoch 338/500
 - 0s - loss: 0.0278 - val_loss: 0.0141
Epoch 339/500
 - 0s - loss: 0.0277 - val_loss: 0.0142
Epoch 340/500
 - 0s - loss: 0.0276 - val_loss: 0.0142
Epoch 341/500
 - 0s - loss: 0.0276 - val_loss: 0.0140
Epoch 342/500
 - 0s - loss: 0.0275 - val_loss: 0.0139
Epoch 343/500
 - 0s - loss: 0.0275 - val_loss: 0.0139
Epoch 344/500
 - 0s - loss: 0.0274 - val_loss: 0.0137
Epoch 345/500
 - 0s - loss: 0.0273 - val_loss: 0.0140
Epoch 346/500
 - 0s - loss: 0.0273 - val_loss: 0.0137
Epoch 347/500
 - 0s - loss: 0.0272 - val_loss: 0.0139
Epoch 348/500
 - 0s - loss: 0.0272 - val_loss: 0.0138
Epoch 349/500
 - 0s - loss: 0.0272 - val_loss: 0.0135
Epoch 350/500
 - 0s - loss: 0.0271 - val_loss: 0.0137
Epoch 351/500
 - 0s - loss: 0.0271 - val_loss: 0.0135
Epoch 352/500
 - 0s - loss: 0.0270 - val_loss: 0.0133
Epoch 353/500
 - 0s - loss: 0.0270 - val_loss: 0.0135
Epoch 354/500
 - 0s - loss: 0.0270 - val_loss: 0.0131
Epoch 355/500
 - 0s - loss: 0.0269 - val_loss: 0.0135
Epoch 356/500
 - 0s - loss: 0.0268 - val_loss: 0.0131
Epoch 357/500
 - 0s - loss: 0.0268 - val_loss: 0.0133
Epoch 358/500
 - 0s - loss: 0.0268 - val_loss: 0.0131
Epoch 359/500
 - 0s - loss: 0.0267 - val_loss: 0.0133
Epoch 360/500
 - 0s - loss: 0.0267 - val_loss: 0.0131
Epoch 361/500
 - 0s - loss: 0.0266 - val_loss: 0.0130
Epoch 362/500
 - 0s - loss: 0.0266 - val_loss: 0.0130
Epoch 363/500
 - 0s - loss: 0.0266 - val_loss: 0.0129
Epoch 364/500
 - 0s - loss: 0.0265 - val_loss: 0.0130
Epoch 365/500
 - 0s - loss: 0.0265 - val_loss: 0.0131
Epoch 366/500
 - 0s - loss: 0.0265 - val_loss: 0.0128
Epoch 367/500
 - 0s - loss: 0.0265 - val_loss: 0.0131
Epoch 368/500
 - 0s - loss: 0.0265 - val_loss: 0.0126
Epoch 369/500
 - 0s - loss: 0.0264 - val_loss: 0.0131
Epoch 370/500
 - 0s - loss: 0.0264 - val_loss: 0.0130
Epoch 371/500
 - 0s - loss: 0.0263 - val_loss: 0.0127
Epoch 372/500
 - 0s - loss: 0.0263 - val_loss: 0.0128
Epoch 373/500
 - 0s - loss: 0.0263 - val_loss: 0.0125
Epoch 374/500
 - 0s - loss: 0.0262 - val_loss: 0.0128
Epoch 375/500
 - 0s - loss: 0.0262 - val_loss: 0.0127
Epoch 376/500
 - 0s - loss: 0.0262 - val_loss: 0.0128
Epoch 377/500
 - 0s - loss: 0.0262 - val_loss: 0.0127
Epoch 378/500
 - 0s - loss: 0.0261 - val_loss: 0.0126
Epoch 379/500
 - 0s - loss: 0.0261 - val_loss: 0.0126
Epoch 380/500
 - 0s - loss: 0.0261 - val_loss: 0.0126
Epoch 381/500
 - 0s - loss: 0.0261 - val_loss: 0.0126
Epoch 382/500
 - 0s - loss: 0.0260 - val_loss: 0.0125
Epoch 383/500
 - 0s - loss: 0.0260 - val_loss: 0.0124
Epoch 384/500
 - 0s - loss: 0.0260 - val_loss: 0.0124
Epoch 385/500
 - 0s - loss: 0.0260 - val_loss: 0.0124
Epoch 386/500
 - 0s - loss: 0.0260 - val_loss: 0.0126
Epoch 387/500
 - 0s - loss: 0.0260 - val_loss: 0.0124
Epoch 388/500
 - 0s - loss: 0.0259 - val_loss: 0.0124
Epoch 389/500
 - 0s - loss: 0.0259 - val_loss: 0.0124
Epoch 390/500
 - 0s - loss: 0.0259 - val_loss: 0.0123
Epoch 391/500
 - 0s - loss: 0.0259 - val_loss: 0.0125
Epoch 392/500
 - 0s - loss: 0.0259 - val_loss: 0.0123
Epoch 393/500
 - 0s - loss: 0.0259 - val_loss: 0.0125
Epoch 394/500
 - 0s - loss: 0.0258 - val_loss: 0.0123
Epoch 395/500
 - 0s - loss: 0.0258 - val_loss: 0.0123
Epoch 396/500
 - 0s - loss: 0.0259 - val_loss: 0.0122
Epoch 397/500
 - 0s - loss: 0.0258 - val_loss: 0.0123
Epoch 398/500
 - 0s - loss: 0.0258 - val_loss: 0.0124
Epoch 399/500
 - 0s - loss: 0.0258 - val_loss: 0.0120
Epoch 400/500
 - 0s - loss: 0.0257 - val_loss: 0.0122
Epoch 401/500
 - 0s - loss: 0.0257 - val_loss: 0.0123
Epoch 402/500
 - 0s - loss: 0.0258 - val_loss: 0.0120
Epoch 403/500
 - 0s - loss: 0.0257 - val_loss: 0.0123
Epoch 404/500
 - 0s - loss: 0.0257 - val_loss: 0.0122
Epoch 405/500
 - 0s - loss: 0.0257 - val_loss: 0.0122
Epoch 406/500
 - 0s - loss: 0.0257 - val_loss: 0.0121
Epoch 407/500
 - 0s - loss: 0.0257 - val_loss: 0.0122
Epoch 408/500
 - 0s - loss: 0.0257 - val_loss: 0.0122
Epoch 409/500
 - 0s - loss: 0.0257 - val_loss: 0.0122
Epoch 410/500
 - 0s - loss: 0.0256 - val_loss: 0.0121
Epoch 411/500
 - 0s - loss: 0.0256 - val_loss: 0.0122
Epoch 412/500
 - 0s - loss: 0.0256 - val_loss: 0.0121
Epoch 413/500
 - 0s - loss: 0.0256 - val_loss: 0.0122
Epoch 414/500
 - 0s - loss: 0.0256 - val_loss: 0.0121
Epoch 415/500
 - 0s - loss: 0.0256 - val_loss: 0.0122
Epoch 416/500
 - 0s - loss: 0.0256 - val_loss: 0.0120
Epoch 417/500
 - 0s - loss: 0.0256 - val_loss: 0.0121
Epoch 418/500
 - 0s - loss: 0.0255 - val_loss: 0.0122
Epoch 419/500
 - 0s - loss: 0.0256 - val_loss: 0.0121
Epoch 420/500
 - 0s - loss: 0.0256 - val_loss: 0.0119
Epoch 421/500
 - 0s - loss: 0.0255 - val_loss: 0.0120
Epoch 422/500
 - 0s - loss: 0.0255 - val_loss: 0.0121
Epoch 423/500
 - 0s - loss: 0.0255 - val_loss: 0.0120
Epoch 424/500
 - 0s - loss: 0.0255 - val_loss: 0.0121
Epoch 425/500
 - 0s - loss: 0.0256 - val_loss: 0.0119
Epoch 426/500
 - 0s - loss: 0.0255 - val_loss: 0.0120
Epoch 427/500
 - 0s - loss: 0.0257 - val_loss: 0.0124
Epoch 428/500
 - 0s - loss: 0.0257 - val_loss: 0.0117
Epoch 429/500
 - 0s - loss: 0.0255 - val_loss: 0.0120
Epoch 430/500
 - 0s - loss: 0.0256 - val_loss: 0.0124
Epoch 431/500
 - 0s - loss: 0.0256 - val_loss: 0.0117
Epoch 432/500
 - 0s - loss: 0.0255 - val_loss: 0.0120
Epoch 433/500
 - 0s - loss: 0.0255 - val_loss: 0.0121
Epoch 434/500
 - 0s - loss: 0.0256 - val_loss: 0.0119
Epoch 435/500
 - 0s - loss: 0.0255 - val_loss: 0.0117
Epoch 436/500
 - 0s - loss: 0.0255 - val_loss: 0.0123
Epoch 437/500
 - 0s - loss: 0.0256 - val_loss: 0.0119
Epoch 438/500
 - 0s - loss: 0.0256 - val_loss: 0.0118
Epoch 439/500
 - 0s - loss: 0.0254 - val_loss: 0.0120
Epoch 440/500
 - 0s - loss: 0.0256 - val_loss: 0.0122
Epoch 441/500
 - 0s - loss: 0.0255 - val_loss: 0.0117
Epoch 442/500
 - 0s - loss: 0.0254 - val_loss: 0.0119
Epoch 443/500
 - 0s - loss: 0.0254 - val_loss: 0.0120
Epoch 444/500
 - 0s - loss: 0.0255 - val_loss: 0.0119
Epoch 445/500
 - 0s - loss: 0.0255 - val_loss: 0.0117
Epoch 446/500
 - 0s - loss: 0.0255 - val_loss: 0.0124
Epoch 447/500
 - 0s - loss: 0.0255 - val_loss: 0.0119
Epoch 448/500
 - 0s - loss: 0.0255 - val_loss: 0.0117
Epoch 449/500
 - 0s - loss: 0.0254 - val_loss: 0.0122
Epoch 450/500
 - 0s - loss: 0.0254 - val_loss: 0.0119
Epoch 451/500
 - 0s - loss: 0.0254 - val_loss: 0.0119
Epoch 452/500
 - 0s - loss: 0.0254 - val_loss: 0.0119
Epoch 453/500
 - 0s - loss: 0.0254 - val_loss: 0.0120
Epoch 454/500
 - 0s - loss: 0.0255 - val_loss: 0.0117
Epoch 455/500
 - 0s - loss: 0.0255 - val_loss: 0.0122
Epoch 456/500
 - 0s - loss: 0.0254 - val_loss: 0.0119
Epoch 457/500
 - 0s - loss: 0.0255 - val_loss: 0.0117
Epoch 458/500
 - 0s - loss: 0.0254 - val_loss: 0.0119
Epoch 459/500
 - 0s - loss: 0.0254 - val_loss: 0.0121
Epoch 460/500
 - 0s - loss: 0.0255 - val_loss: 0.0119
Epoch 461/500
 - 0s - loss: 0.0255 - val_loss: 0.0117
Epoch 462/500
 - 0s - loss: 0.0254 - val_loss: 0.0122
Epoch 463/500
 - 0s - loss: 0.0254 - val_loss: 0.0119
Epoch 464/500
 - 0s - loss: 0.0255 - val_loss: 0.0118
Epoch 465/500
 - 0s - loss: 0.0255 - val_loss: 0.0117
Epoch 466/500
 - 0s - loss: 0.0253 - val_loss: 0.0118
Epoch 467/500
 - 0s - loss: 0.0257 - val_loss: 0.0122
Epoch 468/500
 - 0s - loss: 0.0255 - val_loss: 0.0117
Epoch 469/500
 - 0s - loss: 0.0253 - val_loss: 0.0118
Epoch 470/500
 - 0s - loss: 0.0253 - val_loss: 0.0119
Epoch 471/500
 - 0s - loss: 0.0254 - val_loss: 0.0119
Epoch 472/500
 - 0s - loss: 0.0255 - val_loss: 0.0118
Epoch 473/500
 - 0s - loss: 0.0254 - val_loss: 0.0116
Epoch 474/500
 - 0s - loss: 0.0254 - val_loss: 0.0124
Epoch 475/500
 - 0s - loss: 0.0255 - val_loss: 0.0118
Epoch 476/500
 - 0s - loss: 0.0254 - val_loss: 0.0116
Epoch 477/500
 - 0s - loss: 0.0254 - val_loss: 0.0122
Epoch 478/500
 - 0s - loss: 0.0255 - val_loss: 0.0117
Epoch 479/500
 - 0s - loss: 0.0254 - val_loss: 0.0116
Epoch 480/500
 - 0s - loss: 0.0254 - val_loss: 0.0121
Epoch 481/500
 - 0s - loss: 0.0254 - val_loss: 0.0120
Epoch 482/500
 - 0s - loss: 0.0255 - val_loss: 0.0116
Epoch 483/500
 - 0s - loss: 0.0254 - val_loss: 0.0120
Epoch 484/500
 - 0s - loss: 0.0254 - val_loss: 0.0118
Epoch 485/500
 - 0s - loss: 0.0254 - val_loss: 0.0119
Epoch 486/500
 - 0s - loss: 0.0254 - val_loss: 0.0117
Epoch 487/500
 - 0s - loss: 0.0253 - val_loss: 0.0120
Epoch 488/500
 - 0s - loss: 0.0255 - val_loss: 0.0120
Epoch 489/500
 - 0s - loss: 0.0254 - val_loss: 0.0116
Epoch 490/500
 - 0s - loss: 0.0253 - val_loss: 0.0116
Epoch 491/500
 - 0s - loss: 0.0254 - val_loss: 0.0121
Epoch 492/500
 - 0s - loss: 0.0254 - val_loss: 0.0119
Epoch 493/500
 - 0s - loss: 0.0254 - val_loss: 0.0116
Epoch 494/500
 - 0s - loss: 0.0253 - val_loss: 0.0121
Epoch 495/500
 - 0s - loss: 0.0254 - val_loss: 0.0119
Epoch 496/500
 - 0s - loss: 0.0255 - val_loss: 0.0116
Epoch 497/500
 - 0s - loss: 0.0253 - val_loss: 0.0118
Epoch 498/500
 - 0s - loss: 0.0254 - val_loss: 0.0121
Epoch 499/500
 - 0s - loss: 0.0254 - val_loss: 0.0117
Epoch 500/500
 - 0s - loss: 0.0254 - val_loss: 0.0117
In [174]:
pyplot.plot(history['loss'], label='train')
pyplot.plot(history['val_loss'], label='validation')
pyplot.legend()
pyplot.show()

Test improved model on Validation Data

In [175]:
# make a prediction
%load_ext autoreload
%autoreload 2
import models
inv_yhat, inv_y, rmse=models.make_lstm_prediction(validation_X,validation_y,model,scaler)
print('LSTM Model on Validation Data RMSE: %.3f' % rmse)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
LSTM Model on Validation Data RMSE: 8.860

Test improved model on Unseen(Test) Data

In [176]:
# make a prediction
%load_ext autoreload
%autoreload 2
import models
inv_yhat, inv_y, rmse=models.make_lstm_prediction(test_X,test_y,model,scaler)
print('LSTM Moddel on Test Data RMSE: %.3f' % rmse)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
LSTM Moddel on Test Data RMSE: 8.793